Status

2018-01-22 17:40 UTC

Catarina confirmed Filezilla for SFTP works. Marking CHG as Successful.

2018-01-22 17:19 UTC

Catarina confirmed that 3M image uploads work in the wordpress wui.

She asked to make it possible for 64M uploads. Citing DOS concerns of increasing the query size limit, I highly discouraged this change. Instead, I asked her to test uploading the images via scp or sftp using Filezilla (as we did a training on many months ago). I'm still leaving this CHG open until that's been confirmed as working on her end.

2018-01-18 15:12 UTC

attempted to fix the "Http error" reported by wordpress after attempting to upload a large image
1. using the browser debugger, I saw that it was nginx that returned a 413 error. I fixed this by increasing 'client_max_body_size' to '10M' in /etc/nginx/nginx.conf

[root@hetzner2 dbChange.20180118_12:22:36]# grep -B 1 'client_max_body_size' /etc/nginx/nginx.conf
		# allow large posts for image uploads
		#client_max_body_size 1k;
		#client_max_body_size 900k;
		client_max_body_size 10M;

1. next, I got a 403 error from /wp-admin/async-upload.php
  1. /var/log/httpd/fef.opensourceecology.org/error_log shows a modsecurity issue:

==> /var/log/httpd/fef.opensourceecology.org/error_log <==
[Thu Jan 18 14:56:25.263164 2018] [:error] [pid 27682] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "219"] [id "960915"] [rev "1"] [msg "Multipart parser detected a possible unmatched boundary."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "fef.opensourceecology.org"] [uri "/wp-admin/async-upload.php"] [unique_id "WmC1mU7FUiUY6HdrzSWfWgAAAAA"]

1. 1. as above, whitelisted rule IDs:
    1. 960915, multipart_unmatched_boundary
    2. 200003, multipart_unmatched_boundary

2018-01-18 14:18 UTC

confirmed that a db dump includes image tags with the domain-name hard-coded in the href. that was put in-place by Wordpress's "Add Media" wui button. That's not good; the links should be relative in the db!
1. did some research & found that wp core devs decided 7 years ago to keep absolute paths. This is especially bad for continuous integration or even a basic staging site

* https://core.trac.wordpress.org/ticket/17048

there's no good, robust solution.

* https://stackoverflow.com/questions/17187437/relative-urls-in-wordpress
* https://wordpress.org/plugins/relative-url/
* https://wordpress.org/plugins/root-relative-urls/
* https://deluxeblogtips.com/relative-urls/
* http://www.456bereastreet.com/archive/201010/how_to_make_wordpress_urls_root_relative/

take away is:
1. Let wordpress do its thing. Don't waste effort fighting wp when it auto-inserts an absolute path.
2. However, whenever you have to manually type a path in (ie: when configuring a widget, plugin nav bar, etc), please use a relative link.

2018-01-05 14:05 UTC

Catarina confirmed that the site is working, except for 2 minor issues:

WP inserts absolute-paths in images tags' src attributes when using the wui "Add Media" button
Uploads of large images fails with "Http error"

2018-01-04 22:07 UTC

I did a string string replacement of "http://opensourceecology.org/fef/" with "/" in the entire database.

2018-01-04 15:05 UTC

Catarina pointed out that many old posts still linked to 'http://www.opensourceecology.org/fef', but this was now all broken links.

2018-01-03 20:00 UTC

Fixed the banner icon by changing the hard-coded absolute link in the theme (Appearance -> Menus -> header_menu -> home) from 'http://opensourceecology.org/fef/' to '/'

2018-01-03 19:49 UTC

Marcin responded, indicating that the header icon linked to 'http://www.opensourceecology.org/fef', which linked to 'http://opensourceecology.org/fef-has-visitors/'

2018-01-03 17:52 UTC

Finished change & sent email to service owners for validation.

2018-01-03 17:00 UTC

Began Change

2017-12-21

Catarina validated the ephemeral clone of the existing site running as staging on hetzner 2. She approved downtime for the migration on 2018-01-03 12:00 ET, followed by a 1 week content freeze.

Purpose

This change does the following for fef

entirely migrate the 'fef' wordpress site from hetzner1 (shared hosting) to hetzner2 (dedicated hosting)
changes the domain from 'http://opensourceecology.org/fef/' to 'https://fef.opensourceecology.org'
update core wp from v4.2.18 to v4.9.1
changes the docroot to a subversion repo of the wp codebase, making it easier to update in the future without sacrificing security (ie: permitting wordpress to update itself) for ease-of-use
updates the vhost config to block access to svn files, and more generally: ".*\.(svn|git|hg|bzr|cvs|ht)/.*"
update plugin 'akismet' from v3.1.5 to v4.0.1
update plugin 'cyclone-slider-2' from v2.10.0 to 3.2.0
install & activate plugin 'google-authenticator' to provide 2FA
install & activate plugin 'google-authenticator-encourage-user-activation' to require users to use 2FA
install & activate plugin 'force-strong-passwords' to prevent users from using a poor password, as determined by the built-in wp core password strength feature
install & activate plugin 'rename-wp-login' to obfuscate the wordpress login page (reduces server load & alert emails from HIDS blocking brute force attempts)
install & activate plugin 'vcaching' so wordpress & varnish communicate to clear the cache when needed
updates the vhost config to block all access to ".*wp-login.php"
enable https via nginx
enable cache via varnish

Points of Contact

Change being performed by: Michael Altfield

Service owners: Catarina Mota & Marcin Jakubowski

Apply to Production

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/home/osemain/opensourceecology.org/fef'
dbName_hetzner1='ose_fef'
 dbUser_hetzner1="${mysqlUser_fef}"
 dbPass_hetzner1="${mysqlPass_fef}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}

####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
dbName_hetzner1='ose_fef'
dbName_hetzner2='fef_db'
 dbUser_hetzner2="fef_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/fef.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

# STEP 2: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner2}/current
cp ${backupFileName_db_hetzner1} ${backupFileName_db_hetzner1}.orig

# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql

# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
 time nice mysql -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"

# STEP 3: Add vhost files
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
tar -xzvf ${backupFileName_files_hetzner1}
content_dir=`find ${backupFileName_db_hetzner2} -name wp-content -type d | sort | head -n1`
htaccess_file=`find ${backupFileName_db_hetzner2} -name '.htaccess' -type f | sort | head -n1`
wp_config_file=`find ${backupFileName_db_hetzner2} -name 'wp-config.php' -type f | sort | head -n1`

mkdir -p ${docrootDir_hetzner2}

pushd ${docrootDir_hetzner2}
currentWpVersion=`curl -i https://core.svn.wordpress.org/tags/ | grep '<li>' | tail -n1 | cut -d\" -f2`
svn co https://core.svn.wordpress.org/tags/${currentWpVersion}/ ${docrootDir_hetzner2}
popd

rsync -av --progress ${wp_config_file} ${vhostDir_hetzner2}/
rsync -av --progress ${htaccess_file} ${docrootDir_hetzner2}/
rsync -av --progress ${content_dir} ${docrootDir_hetzner2}/

# make sure this is sudomain, not subdir now
vim ${docrootDir_hetzner2}/.htaccess

# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
# add/replace salts https://api.wordpress.org/secret-key/1.1/salt/
vim ${vhostDir_hetzner2}/wp-config.php

chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0440 "${vhostDir_hetzner2}/wp-config.php"

# INSTALL PLUGINS

# install & configure 2FA plugins
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php
popd

# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate

# install rename-wp-login plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate

# install "SSL Insecure Content Fixer" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate

# install "Varnish Caching" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate

# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
# 3. "Settings" -> "SSL Insecure Content" and then [a] uncheck the "WooCOmmerce" checkbox and [b] change the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"
# 4. "Varnish Caching" and then [a] check the "Enable" checkbox, [b] enter "86400" for the "Homepage cache TTL", [c] enter "86400" for the "Cache TTL", [d] enter "127.0.0.1:6081" for "IPs", [e] check the "Dynamic host" checkbox

CHG-2018-01-03 migrate fef to hetzner2

Contents

Status

2018-01-22 17:40 UTC

2018-01-22 17:19 UTC

2018-01-18 15:12 UTC

2018-01-18 14:18 UTC

2018-01-05 14:05 UTC

2018-01-04 22:07 UTC

2018-01-04 15:05 UTC

2018-01-03 20:00 UTC

2018-01-03 19:49 UTC

2018-01-03 17:52 UTC

2018-01-03 17:00 UTC

2017-12-21

Purpose

Points of Contact

Apply to Production

Navigation menu

CHG-2018-01-03 migrate fef to hetzner2

Status

2018-01-22 17:40 UTC

2018-01-22 17:19 UTC

2018-01-18 15:12 UTC

2018-01-18 14:18 UTC

2018-01-05 14:05 UTC

2018-01-04 22:07 UTC

2018-01-04 15:05 UTC

2018-01-03 20:00 UTC

2018-01-03 19:49 UTC

2018-01-03 17:52 UTC

2018-01-03 17:00 UTC

2017-12-21

Purpose

Points of Contact

Apply to Production

Navigation menu

Search