View source for CHG-2017-09-25 migrate obi to hetzner2

=Status=

==2017-09-25 23:00 UTC==
Change completed successfully. There's a few issues with mixed content warning (and refusals to load) from browsers. Catarina confirmed that she will look into them in the morning. A content freeze (for non-trivial content) will remain in effect until Monday Oct 2nd.

==2017-09-25 21:25 UTC==
Change started after green-light from Catarina.

==2017-09-25 19:41 UTC==
This change is awaiting a content change by Catarina. Once complete, we will have a content freeze for 1 week, and I will begin applying this change to production.

=Purpose=

This change does the following for openbuildinginstitute.org

# update core wp from v4.5.9 to v4.8.1
# changes the docroot to a subversion repo of the wp codebase, making it easier to update in the future without sacrificing security (ie: permitting wordpress to update itself) for ease-of-use
# updates the vhost config to block access to svn files, and more generally: ".*\.(svn|git|hg|bzr|cvs|ht)/.*"
# moves '3Dmodels' dir inside 'wp-content' so it won't be orphaned in future updates/migration
# update plugin 'akismet' from v3.3.3 to v3.3.4
# update plugin 'amr-shortcode-any-widget' from v3.3.3 to v3.6
# update plugin 'duplicate-page' from v2.2 to v2.3
# install & activate plugin 'google-authenticator' to provide 2FA
# install & activate plugin 'google-authenticator-encourage-user-activation' to require users to use 2FA
# install & activate plugin 'force-strong-passwords' to prevent users from using a poor password, as determined by the built-in wp core password strength feature
# install & activate plugin 'rename-wp-login' to obfuscate the wordpress login page (reduces server load & alert emails from HIDS blocking brute force attempts)
# updates the vhost config to block all access to ".*wp-login.php"
# enable https

=Points of Contact=

Change being performed by: [[User:Maltfield|Michael Altfield]]

Service owners: [[User:Catarina|Catarina Mota]] & [[User:Marcin|Marcin Jakubowski]]

=Apply to Production=
<pre>
sudo su -

#################
# set variables #
#################
vhostDir=/var/www/html/openbuildinginstitute.org
dbName=obi_db
dbUser=obi_db_user
 dbPass=CHANGEME
 rootDbPass=CHANGEME

stamp=`date +%Y%m%d_%T`

vhostDirMovedTo="/var/www/html/obi.${stamp}.bak"

################
# make backups #
################

# SYSTEM-SIDE BACKUPS
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log
source /root/backups/backup.settings

# validate
ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*

# LOCAL BACKUPS
tmpDir=/var/tmp/chg.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
service httpd stop

# create backup of all DBs for good measure
 time nice mysqldump -uroot -p$rootDbPass --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz

# dump wp DB contents
 time nice mysqldump -u$dbUser -p$dbPass --database $dbName > $dbName.$stamp.sql

# files backup
rsync -av --progress "/var/www/html/obi" "vhostDir.${stamp}.bak/"

#############
# update db #
#############

# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db"
cp $dbName.$stamp.sql $dbName.$stamp.sql.bak
vim $dbName.$stamp.sql

# update links for 3Dmodels, which we're moving to the 'wp-content' dir
sed -i 's^/3Dmodels/^/wp-content/3Dmodels/^g' "$dbName.$stamp.sql"

 time nice mysql -uroot -p$rootDbPass -sNe "DROP DATABASE IF EXISTS $dbName;" 
 time nice mysql -uroot -p$rootDbPass -sNe "CREATE DATABASE $bName; USE $dbName;"
 time nice mysql -uroot -p$rootDbPass < "$dbName.$stamp.sql"
 time nice mysql -uroot -p$rootDbPass -sNe "GRANT ALL ON $dbName.* TO '$dbUser'@'localhost' IDENTIFIED BY '$dbPass'; FLUSH PRIVILEGES;"

popd

################
# update files #
################

mv "/var/www/html/obi" "${vhostDirMovedTo}"
mkdir -p "${vhostDir}/htdocs"
pushd "${vhostDir}/htdocs"

yum install subversion
svn co https://core.svn.wordpress.org/tags/4.8.1 .

find $vhostDir -type d -exec chmod 750 {} \;
find $vhostDIr -type f -exec chmod 640 {} \;
chown -R apache:apache ${vhostDir}

rsync -av --progress ${vhostDirMovedTo}/wp-config.php ${vhostDir}/
chown apache:apache-admins ${vhostDir}/wp-config.php
chmod 440 ${vhostDir}/wp-config.php

# change WP_HOME, WP_SITEURL, DB_NAME, DB_USER, & DB_PASSWORD as needed
vim ${vhostDir}/wp-config.php

# we want to copy files that don't exist yet in our new install dir. if a file exists in both, don't overwrite the new from the old
rsync -av --progress --ignore-existing "${vhostDirMovedTo}/htdocs/wp-content/" "$vhostDir/htdocs/wp-content/"

# delete unnecessary & empty upgrade dir
rm -rf "${vhostDir}/htdocs/wp-content/upgrade"

# delete unnecessary plugins
rm -rf "${vhostDir}/htdocs/wp-content/plugins/hello.php"
rm -rf "${vhostDir}/htdocs/wp-content/plugins/password-protected"
rm -rf "${vhostDir}/htdocs/wp-content/plugins/simple-301-redirects"

# migrate '3Dmodels' dir, but put it in 'wp-content' for future migrations (the wp upgrade docs migrate this dir already)
rsync -av --progress "${vhostDirMovedTo}/htdocs/3Dmodels" "${vhostDir}/htdocs/wp-content/"

# take the old .htaccess; overwrite existing if needed
rsync -av --progress "${vhostDirMovedTo}/htdocs/.htaccess" "${vhostDir}/htdocs/"

# set permissions, final pass
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${vhostDir}/htdocs/wp-content" -type f -exec chmod 0660 {} \;
find "${vhostDir}/htdocs/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${newVhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"

# update wp plugins
sudo -u wp -i wp --path="${vhostDir}/htdocs" plugin update --all

# update wp themes
#(we skip this in obi's oshine, since it caused issues)
#sudo -u wp -i wp --path="${vhostDir}/htdocs" theme update --all

# install & configure 2FA plugins
sudo -u wp -i wp --path=${vhostDir}/htdocs plugin install google-authenticator --activate
sudo -u wp -i wp --path=${vhostDir}/htdocs plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir}` wp"
cd "${vhostDir}/htdocs/wp-content/plugins/google-authenticator"
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php

# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path="${vhostDir}/htdocs" plugin install force-strong-passwords --activate

# install rename-wp-login plugin
sudo -u wp -i wp --path="${vhostDir}/htdocs" plugin install rename-wp-login --activate

# add deny for 'wp-login.php' _inside_ the VirtualHosts block of the vhost config file
# add deny for svn-related files, and a host of other sensitive dot-files. Again, put this _inside_ the VirtualHosts block
# change vhost file to change to https
mv /etc/httpd/conf.d/obi.conf /etc/httpd/conf.d/obi.conf.$stamp.bak
cat << EOF > /etc/httpd/conf.d/00-openbuildinginstitute.org.conf
<VirtualHost 138.201.84.223:80>
    ServerName openbuildinginstitute.org
    ServerAlias www.openbuildinginstitute.org
    Redirect permanent / https://openbuildinginstitute.org/
</VirtualHost>

Listen openbuildinginstitute.org:443
<VirtualHost openbuildinginstitute.org:443>
        ServerName openbuildinginstitute.org
        ServerAlias www.openbuildinginstitute.org
        DocumentRoot "/var/www/html/openbuildinginstitute.org/htdocs"

        Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org

        # disable mod_security with rules as needed
        # (found by logs in: /var/log/httpd/modsec_audit.log)
        <Location "/wp-admin/">
                <IfModule security2_module>
                        SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109
                </IfModule>
        </Location>

        # block dot files, such as svn files from checking out wp core
        <LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
                Deny From All
        </LocationMatch>

        # block access to 'wp-login.php' from brute-forcers; see wp plugin 'rename-wp-login'
        <LocationMatch .*wp-login.php>
                Deny From All
        </LocationMatch>

</VirtualHost>

<Directory "/var/www/html/openbuildinginstitute.org/htdocs">
        Options -Indexes -Includes
        AllowOverride All
        Require all granted

        Order allow,deny
        Allow from all

        # Harden vhost docroot by blocking all request types except the 3 essentials
        <LimitExcept GET POST HEAD>
                deny from all
        </LimitExcept>

</Directory>

#<LocationMatch .*wp-login.php>                                                  
#  Deny From All                                                                
#</LocationMatch>
vim /etc/httpd/conf.d/TODO


#<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
#   Deny From All
#</LocationMatch>
vim /etc/httpd/conf.d/TODO


vim /etc/httpd/conf.d/TODO

# test apache config
httpd -t

# if OK, apply apache configs
service httpd start

# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
</pre>

=Build Ephemeral Staging=

These commands were used to migrate the existing & outdated wordpress installs to a subversion-backed, ephemeral clone of the site. Changes were iterated to these commands documented here as validation by the site owners exposed issues with the upgrades, missing content, etc. Once the clone was validated entirely, the live site could be brought down & confidently updated per the findings with the ephemeral clone.

==obi==

<pre>
oldVhostDir=/var/www/html/obi
newVhostDir=/var/www/html/obi2
oldDbName=obi_db
newDbName=obi2_db
newDbUser=obi2_user
 newDbPass=CHANGEME
 rootDbPass=CHANGEME

cd
rm -rf $newVhostDir
mkdir -p $newVhostDir/htdocs
pushd $newVhostDir/htdocs

yum install subversion
svn co https://core.svn.wordpress.org/tags/4.8.1 .

find $newVhostDir -type d -exec chmod 750 {} \;
find $newVhostDIr -type f -exec chmod 640 {} \;
chown -R apache:apache $newVhostDir

stamp=`date +%Y%m%d_%T`
tmpDir=/var/tmp/dbChange.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
 time nice mysqldump -uroot -p$rootDbPass --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz
 time nice mysqldump -uroot -p$rootDbPass --databases $oldDbName > $oldDbName.$stamp.sql
cp $oldDbName.$stamp.sql $newDbName.$stamp.sql

# replace the first 2 (non-comment) occurances of $OldDbName with $newDbName
vim $newDbName.$stamp.sql

# update links for 3Dmodels, which we're moving to the 'wp-content' dir
sed -i 's^/3Dmodels/^/wp-content/3Dmodels/^g' "$newDbName.$stamp.sql"

 time nice mysql -uroot -p$rootDbPass -sNe "DROP DATABASE IF EXISTS $newDbName;" 
 time nice mysql -uroot -p$rootDbPass -sNe "CREATE DATABASE $newDbName; USE $newDbName;"
 time nice mysql -uroot -p$rootDbPass < "$newDbName.$stamp.sql"
 time nice mysql -uroot -p$rootDbPass -sNe "GRANT ALL ON $newDbName.* TO '$newDbUser'@'localhost' IDENTIFIED BY '$newDbPass'; FLUSH PRIVILEGES;"

popd

rsync -av --progress $oldVhostDir/wp-config.php $newVhostDir/
chown apache:apache $newVhostDir/wp-config.php
chmod 400 $newVhostDir/wp-config.php

# change WP_HOME, WP_SITEURL, DB_NAME, DB_USER, & DB_PASSWORD
vim $newVhostDir/wp-config.php

# we want to copy files that don't exist yet in our new install dir. if a file exists in both, don't overwrite the new from the old
rsync -av --progress --ignore-existing $oldVhostDir/htdocs/wp-content/ $newVhostDir/htdocs/wp-content/

# delete unnecessary & empty upgrade dir
rm -rf $newVhostDir/htdocs/wp-content/upgrade

# delete unnecessary plugins
rm -rf $newVhostDir/htdocs/wp-content/plugins/hello.php
rm -rf $newVhostDir/htdocs/wp-content/plugins/password-protected
rm -rf $newVhostDir/htdocs/wp-content/plugins/simple-301-redirects

# migrate '3Dmodels' dir, but put it in 'wp-content' for future migrations (the wp upgrade docs migrate this dir already)
rsync -av --progress $oldVhostDir/htdocs/3Dmodels $newVhostDir/htdocs/wp-content/

# take the old .htaccess; overwrite existing if needed
rsync -av --progress $oldVhostDir/htdocs/.htaccess $newVhostDir/htdocs/

# set permissions
chown -R apache:apache "${newVhostDir}"
find "${newVhostDir}" -type d -exec chmod 0750 {} \;
find "${newVhostDir}" -type f -exec chmod 0640 {} \;
find "${newVhostDir}/htdocs/wp-content" -type f -exec chmod 0660 {} \;
find "${newVhostDir}/htdocs/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${newVhostDir}/wp-config.php"
chmod 0440 "${newVhostDir}/wp-config.php"

# update wp plugins
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin update --all

# update wp themes
#sudo -u wp -i wp --path=$newVhostDir/htdocs theme update --all

# install & configure 2FA plugins
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator --activate
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename $newVhostDir` wp"
cd $newVhostDir/htdocs/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php

# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install force-strong-passwords --activate

# install rename-wp-login plugin
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install rename-wp-login --activate

# add vhosts dir to php.ini 'open_basedir'
vim /etc/php.ini

# create vhosts apache conf file w/ deny for 'wp-login.php'
#<LocationMatch .*wp-login.php>                                                  
#  Deny From All                                                                
#</LocationMatch>  

vim /etc/httpd/conf.d/TODO

# apply apache configs
service httpd reload

# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"

</pre>