View source for CHG-2025-XX-XX migrate microfactory to hetzner3

=Status=

==2024-01-29 12:30 UTC==

Initial Ticket draft created on wiki (WIP)

=Change Info=

==Scheduled Time==

This change will take place on 2025-??-?? ??:00 UTC

* = 2025-??-?? ??:00 Kansas City, US
* = 2025-??-?? ??:00 Guayaquil, EC

https://www.timeanddate.com/worldclock/converter.html?iso=20240727T160000&p1=405&p2=1440&p3=93

==Purpose==

This change does the following for microfactory.opensourceecology.org

# entirely migrate the 'microfactory' wordpress site from [[hetzner2]] (EOL CentOS7) to [[hetzner3]] (Debian 12)
# changes site from apache v2.4.6 to v2.4.62
# changes site from varnish v4.0.5 to v7.1.1
# changes site from nginx v1.20.1 to v1.22.1
# changes site from php v5.6 to v8.2
# changes site from using mod_php to php-fpm
# update core wp from v5.1.1 to v6.7.1 
# eliminates subversion from the docroot (which has security issues because the releases are not signed and lacks 3TOFU)
# update plugin 'akismet' from v4.1.1 to v5.3.5
# update plugin 'be-gdpr' from v1.0.0 to v1.1.6
# update plugin 'be-portfolio-post' from v1.1 to v1.1.1
# uninstall plugin 'colorhub'
# uninstall plugin 'force-strong-passwords'
# update plugin 'google-authenticator' from v0.48 to 0.54
# uninstall plugin 'masterslider'
# update plugin 'masterslider' from v3.2.7 to v3.7.8
# update plugin 'meta-box' from v4.17.3 to v5.10.5
# update plugin 'meta-box-conditional-logic' from v1.3 to v1.6.23
# update plugin 'meta-box-show-hide' from v1.0.2 to v1.3.1
# update plugin 'meta-box-tabs' from v1.0.0 to v1.1.18
# update plugin 'oshine-core' from v1.3.4 to v1.6.1
# update plugin 'oshine-modules' from v2.2.5 to v3.3.8
# uninstall plugin 'rename-wp-login'
# update plugin 'revslider' from v5.4.7.4 to 6.7.25
# update plugin 'tatsu' from v2.6.8 to v3.5.3
# uninstall plugin 'typehub'
# update plugin 'vaching' from v1.6.9 to v1.8.3

# install, activate, and configure new plugin 'activitypub' v4.4.0
# install, activate, and configure new plugin 'aurora-heatmap' v1.7.0
# install, activate, and configure new plugin 'melapress-login-security' v2.0.1
# install plugin 'advanced-nocaptcha-recaptcha' v7.5.0
# install plugin 'bulk-media-register' v1.40
# install plugin 'enable-media-replace' v4.1.5
# install plugin 'extensions-leaflet-map' v4.4.3
# install plugin 'hcaptcha-for-forms-and-more' v4.8.0
# install plugin 'include-mastodon-feed' v1.9.9
# install plugin 'leaflet-map' v3.4.1
# install plugin 'raw-html' v1.6.4
# install plugin 'regenerate-thumbnails' v3.1.6
# install plugin 'related-posts-by-taxonomy' v2.7.6
# install plugin 'smart-slider-3' v3.5.1.25
# install plugin 'spam-destroyer' v2.1.4
# install plugin 'wordpress-seo' v24.0
# install plugin 'wp-2fa' v2.8.0
# install plugin 'wpforms-lite' v1.9.2.3
# install plugin 'wpfront-notification-bar' v3.4.2
# install plugin 'wp-pgp-encrypted-emails' v0.8.0
# install plugin 'wp-qrcode' v1.1.1

# update theme 'oshin' from v6.6.4.4 to v7.2.1
# update theme 'twentyeleven' from v3.2 to v4.8
# update theme 'twentyfifteen' from v2.4 to v3.9
# update theme 'twentyfourteen' from v2.6 to v4.1
# update theme 'twentyseventeen' from v2.1 to v3.8
# update theme 'twentysixteen' from v1.9 to v3.4
# update theme 'twentyten' from v2.8 to v4.3
# update theme 'twentythirteen' from v2.8 to v4.3
# update theme 'twentytwelve' from v2.9 to v4.4

==Points of Contact==

Change being performed by: [[User:Maltfield|Michael Altfield]]

Service owners: [[User:Catarina|Catarina Mota]] & [[User:Marcin|Marcin Jakubowski]]

==Time Length==

This whole process is expected to take 1-3 hours.

Some systems could be impacted for days, if issues are encountered.

==Systems Impacted==

This change impacts both hetzner2 and hetzner3. It's possible that we'll need to restart one or both of these servers during this migration, which could affect the availability of all OSE & OBI websites.

==Staging Test==

n/a

=Change Steps=
<pre>
####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
source /root/backups/backup.settings
/root/backups/backup.sh

# when finished, verify that the whole system backup was successful before proceeding
sudo rclone lsl b2:ose-server-backups | grep -i $(date +%Y%m%d)

# DECLARE VARIABLES
vhost_name='microfactory.opensourceecology.org'
dbName='microfactory_db'
 dbUser="CHANGEME"
 dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"

# STEP 1: Bring site down on hetzner2
# uncomment the SITE_DOWN section
vim /etc/nginx/conf.d/${vhost_name}.conf

nginx -t
# only run this next command if the above command was successful
service nginx reload

# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/

time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}

# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}

# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/

####################
# run on hetzner3 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, verify that the whole system backup was successful before proceeding
rclone lsl b2:ose-server-backups | grep -i $(date +%Y%m%d)

# DECLARE VARIABLES
vhost_name='microfactory.opensourceecology.org'
dbName='microfactory_db'
 dbUser="CHANGEME"
 dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"

# STEP 1: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig

# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner2} > db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
 time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"

# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mv var/www/html/${vhost_name} ${vhostDir}

# remove '.svn' dirs (we no longer use svn, for security)
find ${docrootDir} -iname '.svn' -exec rm -rf '{}' \;

# add wordpress bug fix
# is the bug fix already present?
if [[ ! $(grep 'https://core.trac.wordpress.org/ticket/48693' ${vhostDir}/wp-config.php) ]]; then
	# the bug fix is absent; add it

	backup_filename="wp-config.`date "+%Y%m%d_%H%M%S"`.php"
	mv ${vhostDir}/wp-config.php ${vhostDir}/${backup_filename}

	cat > ${vhostDir}/wp-config.php <<'EOF'
<?php

# fix wordpress bugs
# * https://core.trac.wordpress.org/ticket/48693
# * https://core.trac.wordpress.org/ticket/62693
if( ! function_exists('ini_set') ){
        function ini_set(){
                return;
        }
}
if( ! function_exists('chmod') ){
        function chmod(){
                return;
        }
}

EOF

	tail -n +2 ${vhostDir}/${backup_filename} >> ${vhostDir}/wp-config.php

fi

# verify
ls
vim ${vhostDir}/wp-config.php

# UPDATE CORE

rsync -av --progress /var/tmp/wordpress/core/wordpress/ ${docrootDir}

# UPDATE OLD PLUGINS

for plugin_path in $(find "${docrootDir}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
        plugin=$(basename "${plugin_path}")
        source_path="/var/tmp/wordpress/plugins/${plugin}"
        
        echo "${plugin}"
        rm -rf ${plugin_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${plugin_path}/"
        fi
done

# INSTALLL NEW PLUGINS

new_plugins="activitypub aurora-heatmap melapress-login-security wps-hide-login raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"

for plugin in ${new_plugins}; do
        plugin_path="${docrootDir}/wp-content/plugins/${plugin}"
        source_path="/var/tmp/wordpress/plugins/${plugin}"
        
        if [ -d "${source_path}" ]; then
                echo "${plugin}"
                rm -rf ${plugin_path};
                rsync -a ${source_path}/ "${plugin_path}/"
        fi
done

# delete Master Slider plugin (MS_Aq_Resize.process() error: Image file does not exist)
rm -rf ${docrootDir}/wp-content/plugins/masterslider*

# UPDATE/INSTALL THEMES

for theme_path in $(find "${docrootDir}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	source_path="/var/tmp/wordpress/themes/${theme}"
	
	echo "${theme}"
	rm -rf ${theme_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${theme_path}/"
	fi
done

# SET PERMISSIONS
time /usr/local/bin/fix_web_permissions.sh

# ACTIVATE NEW PLUGINS

activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
	sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done

# configure melapress-login-security
echo 'DELETE FROM wp_options WHERE option_name = "mls_options";' | mysql -u${dbUser} -p${dbPass} ${dbName}
echo 'DELETE FROM wp_options WHERE option_name = "mls_setting";' | mysql -u${dbUser} -p${dbPass} ${dbName}

cat << EOF | mysql -u${dbUser} -p${dbPass} ${dbName}
INSERT INTO wp_options (option_name, option_value) VALUES ('mls_options', 'a:62:{s:13:"master_switch";s:3:"yes";s:26:"activate_password_policies";s:3:"yes";s:37:"activate_password_expiration_policies";s:2:"no";s:34:"activate_password_recycle_policies";s:2:"no";s:16:"enforce_password";s:2:"no";s:10:"min_length";s:2:"20";s:16:"password_history";s:1:"1";s:16:"inherit_policies";s:2:"no";s:15:"password_expiry";a:2:{s:5:"value";s:1:"0";s:4:"unit";s:6:"months";}s:8:"ui_rules";a:7:{s:7:"history";s:2:"no";s:8:"username";s:2:"no";s:6:"length";b:1;s:7:"numeric";s:2:"no";s:8:"mix_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:5:"rules";a:6:{s:6:"length";b:1;s:7:"numeric";s:2:"no";s:10:"upper_case";s:2:"no";s:10:"lower_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:23:"change_initial_password";s:2:"no";s:12:"timed_logins";s:2:"no";s:21:"timed_logins_schedule";a:7:{s:6:"monday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:7:"tuesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:9:"wednesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"thursday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"friday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"saturday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"sunday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}}s:22:"inactive_users_enabled";s:2:"no";s:21:"inactive_users_expiry";a:2:{s:5:"value";i:30;s:4:"unit";s:4:"days";}s:30:"inactive_users_reset_on_unlock";s:2:"no";s:29:"failed_login_policies_enabled";s:3:"yes";s:21:"failed_login_attempts";s:1:"5";s:27:"failed_login_reset_attempts";s:4:"1440";s:27:"failed_login_unlock_setting";s:5:"timed";s:24:"failed_login_reset_hours";s:2:"60";s:29:"failed_login_reset_on_unblock";s:2:"no";s:18:"disable_self_reset";s:3:"yes";s:26:"disable_self_reset_message";b:0;s:27:"deactivated_account_message";s:159:"Your WordPress user has been deactivated. Please contact the <a href="mailto:michael@michaelaltfield.net">website administrator</a> to activate back your user.";s:19:"timed_login_message";b:0;s:30:"locked_user_disable_self_reset";s:2:"no";s:38:"locked_user_disable_self_reset_message";b:0;s:17:"restrict_login_ip";s:2:"no";s:23:"restrict_login_ip_count";i:3;s:22:"restrict_login_message";s:0:"";s:22:"notify_password_expiry";s:2:"no";s:30:"notify_password_reset_on_login";b:0;s:27:"notify_password_expiry_days";i:0;s:27:"notify_password_expiry_unit";s:4:"days";s:26:"restrict_login_credentials";s:7:"default";s:34:"restrict_login_credentials_message";s:0:"";s:24:"enable_sessions_policies";b:0;s:23:"remember_session_expiry";a:2:{s:5:"value";i:14;s:4:"unit";s:4:"days";}s:22:"default_session_expiry";a:2:{s:5:"value";i:2;s:4:"unit";s:4:"days";}s:22:"enable_device_policies";b:0;s:35:"enable_device_policies_admin_alerts";s:2:"no";s:25:"enable_security_questions";b:0;s:17:"enabled_questions";a:0:{}s:36:"device_policies_prompt_email_content";s:0:"";s:41:"device_policies_admin_alert_email_content";s:0:"";s:36:"device_policies_prompt_email_subject";s:0:"";s:41:"device_policies_admin_alert_email_subject";s:0:"";s:25:"min_answered_needed_count";i:3;s:39:"password_reset_request_disabled_message";s:0:"";s:24:"password_expired_message";s:0:"";s:36:"inactive_user_account_locked_message";s:0:"";s:51:"inactive_user_account_locked_reset_disabled_message";s:0:"";s:38:"restrict_logins_prompt_failure_message";s:0:"";s:34:"timed_logins_login_blocked_message";s:0:"";s:39:"restrict_login_ip_login_blocked_message";s:0:"";s:35:"failed_logins_login_blocked_message";s:0:"";s:40:"security_prompt_response_failure_message";s:0:"";s:24:"timed_logins_auto_logout";b:0;s:13:"ppm-user-role";s:0:"";s:22:"excluded_special_chars";s:0:"";}');
EOF

cat << EOF | mysql -u${dbUser} -p${dbPass} ${dbName}
INSERT INTO wp_options (option_name, option_value) VALUES ('mls_setting', 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}');
EOF
</pre>

TODO: Update DNS to point to hetzner3

# wait until DNS changes to hetzner3
<pre>
while true; do date; dig store.opensourceecology.org; sleep 10; echo; done
</pre>

Finally, log into the new wordpress site (proceed with any db upgrades it walks you through), and update settings:

# "Media" -> "Library"
## Upload https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v3-1blarge.jpg
## Upload https://wiki.opensourceecology.org/wiki/File:1day.jpg
# "Settings" -> "ActivityPub" -> "Settings" -> "Enable profiles by type" = "Blog profile only"
# "Settings" -> "ActivityPub" -> "Settings" -> "Blog-Profile" -> "Change profile ID" = "ose"
# "Settings" -> "ActivityPub" -> "Settings" -> "Blog-Profile" -> "Change Header Image" = Select "design.jpg", cropped such the top just includes the whole OSHW logo on the hat

# quick confirmation of 'melapress-login-security'
# "Login Security" -> "Login Security Policies"
# Make sure password polices are enabled, and >20 chars are required
# "Login Security" -> "Login page hardening"
# Make sure "Login page URL" is set to "ose-hidden-login"

# "Settings" -> "Google Authenticator" -> Check every box under "Roles requiring Google Authenticator Enabled"
# Click "Save Changes" Button

==Validation Steps==

# Load the website in your web browser. Make sure it looks sane
# Login to the website
# Verify that the wordpress admin WUI states that the current version is v6.6.1
# Attempt to make some trivial change
# Confirm that the change is publicly visible on the website
# Revert the trivial change
# Confirm that the change has been reverted on the website
# search for 'https://microfacotry.opensourceecology.org/@ose' and 'ose@microfactory.opensourceecology.org' on some mastodon site and see if it's working

==Revert Steps==

TODO

=See Also=

# [[CHG-2018-02-05_migrate_osemain_to_hetzner2]] Last wordpress migration from hetzner1 to hetzner2
# [[CHG-2024-07-26 yum update]] Last (possible) update to hetzner2
# [[CHG-2025-XX-XX_migrate_store_to_hetzner3]]
# [[:Category: CHGs|List of other CHG "tickets"]]

[[Category: CHGs]]