Status

2024-12-11 09:52 UTC

Initial Ticket draft created on wiki (WIP)

Change Info

Scheduled Time

This change will take place on 2025-??-?? ??:00 UTC

= 2025-??-?? ??:00 Kansas City, US
= 2025-??-?? ??:00 Guayaquil, EC

https://www.timeanddate.com/worldclock/converter.html?iso=20240727T160000&p1=405&p2=1440&p3=93

Purpose

This change does the following for store.opensourceecology.org

entirely migrate the 'store' wordpress site from hetzner2 (EOL CentOS7) to hetzner3 (Debian 12)
changes site from apache v2.4.6 to v2.4.62
changes site from varnish v4.0.5 to v7.1.1
changes site from nginx v1.20.1 to v1.22.1
changes site from php v5.6 to v8.2
changes site from using mod_php to php-fpm
update core wp from v5.1.1 to v6.7.1
eliminates subversion from the docroot (which has security issues because the releases are not signed and lacks 3TOFU)
update plugin 'akismet' from v4.1.1 to v5.3.3
update plugin 'be-gdpr' from v1.0.0 to v1.1.6
uninstall plugin 'be-portfolio-post'
update plugin 'classic-editor' from v1.4 to v1.6.5
uninstall plugin 'colorhub'
update plugin 'contact-form-7' from v5.1.1 to v5.9.8
uninstall plugin 'force-strong-passwords'
update plugin 'google-authenticator' from v0.48 to 0.54
uninstall plugin 'masterslider'
update plugin 'meta-box' from v4.17.3 to v5.10.5
update plugin 'meta-box-conditional-logic' from v1.3 to v1.6.23
update plugin 'meta-box-show-hide' from v1.0.2 to v1.3.1
update plugin 'meta-box-tabs' from v1.0.0 to v1.1.18
update plugin 'oshine-core' from v1.3.4 to v1.6.1
update plugin 'oshine-modules' from v2.2.5 to v3.3.8
uninstall plugin 'redux-vendor-support'
uninstall plugin 'rename-wp-login'
update plugin 'revslider' from v5.4.7.4 to 6.7.25
update plugin 'tatsu' from v2.6.8 to v3.5.3
uninstall plugin 'typehub'
update plugin 'vaching' from v1.6.9 to v1.8.3
update plugin 'woocommerce' from v3.5.7 to v9.5.1
update plugin 'coingate-for-woocommerce' from v1.2.2 to v2.1.1

install, activate, and configure new plugin 'activitypub' v4.4.0
install, activate, and configure new plugin 'aurora-heatmap' v1.7.0
install, activate, and configure new plugin 'melapress-login-security' v2.0.1
install plugin 'advanced-nocaptcha-recaptcha' v7.5.0
install plugin 'bulk-media-register' v1.40
install plugin 'enable-media-replace' v4.1.5
install plugin 'extensions-leaflet-map' v4.4.3
install plugin 'hcaptcha-for-forms-and-more' v4.8.0
install plugin 'include-mastodon-feed' v1.9.9
install plugin 'leaflet-map' v3.4.1
install plugin 'raw-html' v1.6.4
install plugin 'regenerate-thumbnails' v3.1.6
install plugin 'related-posts-by-taxonomy' v2.7.6
install plugin 'smart-slider-3' v3.5.1.25
install plugin 'spam-destroyer' v2.1.4
install plugin 'woocommerce-gateway-stripe' v9.0.0
install plugin 'woocommerce-multilingual' v5.3.9
install plugin 'wordpress-seo' v24.0
install plugin 'wp-2fa' v2.8.0
install plugin 'wpforms-lite' v1.9.2.3
install plugin 'wpfront-notification-bar' v3.4.2
install plugin 'wp-pgp-encrypted-emails' v0.8.0
install plugin 'wp-qrcode' v1.1.1

change theme from 'oshin' to 'twentyseventeen'
update theme 'oshin' from v6.6.4.4 to v7.2.1
update theme 'storefront' from v2.4.5 to v4.6.0
update theme 'twentyeleven' from v3.2 to v4.8
update theme 'twentyfifteen' from v2.4 to v3.9
update theme 'twentyfourteen' from v2.6 to v4.1
update theme 'twentynineteen' from v1.3 to v2.9
update theme 'twentyseventeen' from v2.1 to v3.8
update theme 'twentysixteen' from v1.9 to v3.4
update theme 'twentyten' from v2.8 to v4.3
update theme 'twentythirteen' from v2.8 to v4.3
update theme 'twentytwelve' from v2.9 to v4.4

Points of Contact

Change being performed by: Michael Altfield

Service owners: Catarina Mota & Marcin Jakubowski

Time Length

This whole process is expected to take 1-3 hours.

Some systems could be impacted for days, if issues are encountered.

Systems Impacted

This change impacts both hetzner2 and hetzner3. It's possible that we'll need to restart one or both of these servers during this migration, which could affect the availability of all OSE & OBI websites.

Staging Test

n/a

Change Steps

####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
source /root/backups/backup.settings
/root/backups/backup.sh

# when finished, verify that the whole system backup was successful before proceeding
sudo rclone lsl b2:ose-server-backups | grep -i $(date +%Y%m%d)

# DECLARE VARIABLES
vhost_name='store.opensourceecology.org'
dbName='store_db'
 dbUser="CHANGEME"
 dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"

# STEP 1: Bring site down on hetzner2
# uncomment the SITE_DOWN section
vim /etc/nginx/conf.d/${vhost_name}.conf

nginx -t
# only run this next command if the above command was successful
service nginx reload

# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/

time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}

# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}

# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/

####################
# run on hetzner3 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, verify that the whole system backup was successful before proceeding
rclone lsl b2:ose-server-backups | grep -i $(date +%Y%m%d)

# DECLARE VARIABLES
vhost_name='store.opensourceecology.org'
dbName='store_db'
 dbUser="CHANGEME"
 dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"

# STEP 1: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig

# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner2} > db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
 time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"

# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mv var/www/html/${vhost_name} ${vhostDir}

# remove '.svn' dirs (we no longer use svn, for security)
find ${docrootDir} -iname '.svn' -exec rm -rf '{}' \;

# add wordpress bug fix
# is the bug fix already present?
if [[ ! $(grep 'https://core.trac.wordpress.org/ticket/48693' ${vhostDir}/wp-config.php) ]]; then
	# the bug fix is absent; add it

	backup_filename="wp-config.`date "+%Y%m%d_%H%M%S"`.php"
	mv ${vhostDir}/wp-config.php ${vhostDir}/${backup_filename}

	cat > ${vhostDir}/wp-config.php <<'EOF'
<?php

# fix wordpress bugs
# * https://core.trac.wordpress.org/ticket/48693
# * https://core.trac.wordpress.org/ticket/62693
if( ! function_exists('ini_set') ){
        function ini_set(){
                return;
        }
}
if( ! function_exists('chmod') ){
        function chmod(){
                return;
        }
}

EOF

	tail -n +2 ${vhostDir}/${backup_filename} >> ${vhostDir}/wp-config.php

fi

# verify
ls
vim ${vhostDir}/wp-config.php

# UPDATE CORE

rsync -av --progress /var/tmp/wordpress/core/wordpress/ ${docrootDir}

# UPDATE OLD PLUGINS

for plugin_path in $(find "${docrootDir}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
        plugin=$(basename "${plugin_path}")
        source_path="/var/tmp/wordpress/plugins/${plugin}"
        
        echo "${plugin}"
        rm -rf ${plugin_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${plugin_path}/"
        fi
done

# INSTALLL NEW PLUGINS

new_plugins="activitypub aurora-heatmap melapress-login-security wps-hide-login raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"

for plugin in ${new_plugins}; do
        plugin_path="${docrootDir}/wp-content/plugins/${plugin}"
        source_path="/var/tmp/wordpress/plugins/${plugin}"
        
        if [ -d "${source_path}" ]; then
                echo "${plugin}"
                rm -rf ${plugin_path};
                rsync -a ${source_path}/ "${plugin_path}/"
        fi
done

# UPDATE/INSTALL THEMES

for theme_path in $(find "${docrootDir}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	source_path="/var/tmp/wordpress/themes/${theme}"
	
	echo "${theme}"
	rm -rf ${theme_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${theme_path}/"
	fi
done

# SET PERMISSIONS

# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;

#############
# WORDPRESS #
#############

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do

	wp_docroot="$(dirname "${wordpress_site}")"
	vhost_dir="$(dirname "${wp_docroot}")"

	chown -R not-apache:www-data "${vhost_dir}"
	find "${vhost_dir}" -type d -exec chmod 0050 {} \;
	find "${vhost_dir}" -type f -exec chmod 0040 {} \;

	chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
	chmod 0040 "${vhost_dir}/wp-config.php"

	[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
	find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;

	[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
	find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;

done

###########
# phpList #
###########

phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"

for vhost_dir in $phplist_sites; do
 
	for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
	for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
	for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
 
	for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
	for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
	for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
	for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done

done

# ACTIVATE NEW PLUGINS

activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
	sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done

TODO: Update DNS to point to hetzner3

wait until DNS changes to hetzner3

while true; do date; dig store.opensourceecology.org; sleep 10; echo; done

Finally, log into the new wordpress site (proceed with any db upgrades it walks you through), and update settings:

"Appearance" -> Themes. Activate "Twenty Seventeen"
"Settings" -> "Reading" -> "Homepage" = "Sample Page"
"Media" -> "Library"
1. Upload https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v3-1blarge.jpg
2. Upload https://wiki.opensourceecology.org/wiki/File:1day.jpg
"Settings" -> "ActivityPub" -> "Settings" -> "Enable profiles by type" = "Blog profile only"
"Settings" -> "ActivityPub" -> "Settings" -> "Blog-Profile" -> "Change profile ID" = "ose"
"Settings" -> "ActivityPub" -> "Settings" -> "Blog-Profile" -> "Change Header Image" = Select "1day.jpg", cropped such the bottom is exactly the bottom of Catarina's white coat
"Settings" -> "General" -> "Choose a Site Icon" -> Select "OSE-logo-blueprint-bg-v3-1blarge.jpg", cropped such that there is only a small buffer on the left & right of the text

"Login Security" -> "Login Security Policies" -> tick the box that said "enable login security policies"
"Login Security" -> "Login Security Policies" -> tick the box that said "Activate password policies"
"Login Security" -> "Login Security Policies" -> change "Passwords must be X characters minimum" to "20"
"Login Security" -> "Login Security Policies" -> uncheck "Password must contain at least one uppercase and one lowercase character. "
"Login Security" -> "Login Security Policies" -> uncheck "Password must contain at least one numeric character (0-9)."
"Login Security" -> "Login Security Policies" -> uncheck "Password must contain at least one special character, i.e., a character that is not a letter or a umber, such as ( , ? € ! @ # * etc"
"Login Security" -> "Login Security Policies" -> check "Reset password on first login "
"Login Security" -> "Login Security Policies" -> check "Do not send password reset links "
"Login Security" -> "Login Security Policies" -> check "Activate failed login policies "
"Login Security" -> "Login Security Policies" -> change "When a user is locked" from "it can be only unlocked by the administrator" to "unlock it after 60 minutes"
"Login Security" -> "Login Security Policies" -> uncheck "Require blocked users to reset password on unblock. "
Click the "Save Changes" button
"Login Security" -> "Login page hardening" -> in the input form next to "Login page URL", I enter "ose-hidden-login"
Click the "Save Changes" button

"Settings" -> "Google Authenticator" -> Check every box under "Roles requiring Google Authenticator Enabled"
Click "Save Changes" Button

Validation Steps

Load the website in your web browser. Make sure it looks sane
Login to the website
Verify that the wordpress admin WUI states that the current version is v6.6.1
Attempt to make some trivial change
Confirm that the change is publicly visible on the website
Revert the trivial change
Confirm that the change has been reverted on the website
search for 'https://store.opensourceecology.org/@opensourceeverything' and 'opensourceeverything@store.opensourceecology.org' on some mastodon site and see if it's working

Revert Steps

TODO

CHG-2025-XX-XX migrate store to hetzner3

Contents

Status

2024-12-11 09:52 UTC

Change Info

Scheduled Time

Purpose

Points of Contact

Time Length

Systems Impacted

Staging Test

Change Steps

Validation Steps

Revert Steps

See Also

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Open Source Ecology

Tools