CHG-2025-XX-XX migrate wiki to hetzner3

Status

2024-12-29 11:06 UTC

Initial Ticket draft created on wiki (WIP)

Change Info

Scheduled Time

This change will take place on 2025-??-?? ??:00 UTC

• = 2025-??-?? ??:00 Kansas City, US
• = 2025-??-?? ??:00 Guayaquil, EC

https://www.timeanddate.com/worldclock/converter.html?iso=20240727T160000&p1=405&p2=1440&p3=93

Purpose

This change does the following for wiki.opensourceecology.org

1. entirely migrate the OSE MediaWiki site from hetzner2 (EOL CentOS7) to hetzner3 (Debian 12)
2. changes site from apache v2.4.6 to v2.4.62
3. changes site from varnish v4.0.5 to v7.1.1
4. changes site from nginx v1.20.1 to v1.22.1
5. changes site from php v5.6 to v8.2.26
6. changes site from using mod_php to php-fpm
7. changes site from mariadb v5.5.68 to v10.11.6
8. update ICU from v50.1.2
9. update core MediaWiki from v1.30.0 to v1.43.0

1. update "Cologne Blue" skin from unknown version to 9134d2e
2. update "Modern" skin from unknown version to 5597681
3. update "MonoBook" skin from unknown version to unknown version
4. update "Vector" skin from unknown version to v1.0.0

1. update "Confirm User Accounts" extension from version 4fe25f7 to bb470fd
2. update "Interwiki" extension from version v3.1 20160307 to v3.2
3. update "Nuke" extension from version 1.3.0 to unknown version
4. update "Replace Text" extension from version v1.2 (4426752) to v1.8
5. update "UserMerge" extension from version 1.10.1 (4546537) to v1.10.2 (f9d2664)
6. update "CategoryTree" extension from version 850c018 to unknown version
7. update "Cite" extension from unknown version to unknown version
8. update "ParserFunctions" extension from v1.6.0 to v1.6.1
9. update "Widgets" extension from version v1.3.0 (fce5acc) to v1.6.0 (50da5c6)
10. update "ConfirmEdit" extension from v1.5.0 to v1.6.0
11. install "hCaptcha" extension (unknown version)
12. uninstall "reCaptcha" extension
13. changes ConfirmAccount CAPTCHA from SimpleCaptcha (trivial math) to hCaptcha
14. update "Gadgets" extension from unknown version to unknown version
15. update "OATHAuth" extension from version v0.2.2 (bed2e4b) to v0.5.0

Points of Contact

Change being performed by: Michael Altfield

Service owners: Marcin Jakubowski

Time Length

This whole process is expected to take 6-24 hours.

Some systems could be impacted for days, if issues are encountered.

Systems Impacted

This change impacts both hetzner2 and hetzner3. It's possible that we'll need to restart one or both of these servers during this migration, which could affect the availability of all OSE & OBI websites.

Staging Test

n/a

Change Steps

# TODO: actually verify these commands

####################
# run on hetzner2 #
####################

sudo su -

# DECLARE VARIABLES
vhost_name='wiki.opensourceecology.org'
dbName='osewiki_db'
 dbUser="CHANGEME"
 dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner2/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"

# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/

time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}

# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}

# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/

####################
# run on hetzner3 #
####################

sudo su -

# DECLARE VARIABLES
vhost_name='wiki.opensourceecology.org'
dbName='osewiki_db'
 dbUser="CHANGEME"
 dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
wgCaptchaSecret=$(cat /dev/random | strings | grep -oE '[A-Za-z0-9]' | perl -p -e 's/\n//g' | head -c 64)
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"
chg_dir="/var/tmp/CHG_${stamp}_wiki_1.30-to-1.35"
mkdir -p ${chg_dir}/{pre,post}

# STEP 1: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig

# extract .sql.bz2 -> .sql
time nice bzip2 -dc ${backupFileName_db_hetzner2} > db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
 time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"

# STEP 2: Add vhost files
mv "${vhostDir}" "${chg_dir}/pre/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}

mkdir -p ${vhostDir}
rsync -av --progress /var/tmp/mediawiki/mediawiki-1.35.0/ ${docrootDir}/

rsync -av --progress var/www/html/wiki.opensourceecology.org/LocalSettings.php ${vhostDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php ${docrootDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/htdocs/images ${docrootDir}/

# backup LocalSettings.php
cp ${vhostDir}/LocalSettings.php ${vhostDir}/LocalSettings.$(date "+%Y%m%d_%H%M%S").php

# fix $wgNamespacesWithSubpages
sed -i 's%^\(\s*\)[^#]*$wgNamespacesWithSubpages\(.*\)%\1#$wgNamespacesWithSubpages\2%' ${vhostDir}/LocalSettings.php

grep '$wgNamespacesWithSubpages = \[' ${vhostDir}/LocalSettings.php || sed -i 's%^\(\s*\)#$wgNamespacesWithSubpages\[NS_IMAGE\]\(.*\)%\1#$wgNamespacesWithSubpages\[NS_IMAGE\]\2\n$wgNamespacesWithSubpages = [\n      NS_MAIN => true,\n      NS_TEMPLATE => true,\n      NS_CATEGORY => true,\n      NS_MEDIA => true,\n#      NS_IMAGE => true,\n];%' ${vhostDir}/LocalSettings.php

# deprecate require_once() for extensions (use wfLoadExtension() after upgrades instead)
# * https://www.mediawiki.org/wiki/Manual:Upgrading#Upgrade_extensions
sed -i 's%^\(\s*\)[^#]*require_once\(.*\)/extensions/\(.*\)%\1#require_once\2/extensions/\3%' ${vhostDir}/LocalSettings.php

# deprecate require_once() for skins (use wfLoadSkin() after upgrades instead)
# * https://www.mediawiki.org/wiki/Manual:Upgrading#Skin_registration
sed -i 's%^\(\s*\)[^#]*require_once\(.*\)/skins/\(.*\)%\1#require_once\2/skins/\3%' ${vhostDir}/LocalSettings.php

# remove require_once() for DefaultSettings.php, which was deprecated in v1.39
#  * https://www.mediawiki.org/w/index.php?title=Defaultsettings.php
sed -i 's%^\(\s*\)[^#]*require_once\(.*\)/DefaultSettings.php\(.*\)%\1#require_once\2/DefaultSettings.php\3%' ${vhostDir}/LocalSettings.php

# add $wgExtensionDirectory (defaults to null in MediaWiki >= v1.37)
#  * https://www.mediawiki.org/wiki/Manual:$wgExtensionDirectory
grep '$wgExtensionDirectory' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's/# EXTENSIONS #\n##############/# EXTENSIONS #\n##############\n\$wgExtensionDirectory = "\$IP\/extensions";/igs' ${vhostDir}/LocalSettings.php

# don't yet load UserMerge (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)UserMerge\(.*\)%\1#wfLoadExtension\2UserMerge\3%' ${vhostDir}/LocalSettings.php

# don't yet load Renameuser (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Renameuser\(.*\)%\1#wfLoadExtension\2Renameuser\3%' ${vhostDir}/LocalSettings.php

# don't yet load ConfirmEdit (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtensions\(.*\)ConfirmEdit\(.*\)%\1#wfLoadExtensions\2ConfirmEdit\3%' ${vhostDir}/LocalSettings.php

# uncomment wgServer (now defaults to 'false') https://www.mediawiki.org/wiki/Manual:$wgServer#Default_value
sed -i 's%^\(\s*\)#$wgServer\(.*\)%\1$wgServer\2%' ${vhostDir}/LocalSettings.php

# fix $wgServer
grep '#$wgServer' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%\$wgServer = "http://opensourceecology.org";
%#\$wgServer = "http://opensourceecology.org";\n\$wgServer = "https://wiki.opensourceecology.org";\n%igs' ${vhostDir}/LocalSettings.php

# fix MediaWiki bugs
grep 'ini_set' ${vhostDir}/LocalSettings.php || sed -i 's%^<?php%<?php\n# fix mediawiki bugs\n# * https://phabricator.wikimedia.org/T385965\nif( ! function_exists("ini_set") ){\n\tfunction ini_set(){\n\t\treturn;\n\t}\n}\nif( ! function_exists("putenv") ){\n\tfunction putenv(){\n\t\treturn;\n\t}\n}\nif( ! function_exists("php_uname") ){\n\tfunction php_uname(){\n\t\treturn;\n\t}\n}\nif( ! function_exists("set_time_limit") ){\n\tfunction set_time_limit(){\n\t\treturn;\n\t}\n}\nif( ! function_exists("chmod") ){\n\tfunction chmod(){\n\t\treturn;\n\t}\n}\n%' ${vhostDir}/LocalSettings.php

# FIX PERMISSIONS
time /usr/local/bin/fix_web_permissions.sh

# UPGRADE MEDIAWIKI TO v1.35.0
time nice sudo -u www-data php "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.35.log
time nice sudo -u www-data php "${docrootDir}/maintenance/populateContentTables.php" &> ${chg_dir}/populateContentTables-to-v1.35.log
time nice sudo -u www-data php "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.35b.log

# clear varnish cache (before verify step below)
varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'

# verify by loading:
#
# * https://wiki.opensourceecology.org/wiki/Special:Version
#
# make sure the MediaWiki version specified is v1.35.0

# UPGRADE MEDIAWIKI TO v1.43.0
chg_dir="/var/tmp/CHG_${stamp}_wiki_1.35-to-1.43"
mkdir -p ${chg_dir}/{pre,post}

# Add vhost files
vhost_backup_path="${chg_dir}/pre/${vhost_name}.${stamp}"
mv "${vhostDir}" "${vhost_backup_path}"

mkdir -p ${vhostDir}
rsync -av --progress /var/tmp/mediawiki/mediawiki-1.43.0/ ${docrootDir}/

rsync -av --progress ${vhost_backup_path}/LocalSettings.php* ${vhostDir}/
rsync -av --progress ${vhost_backup_path}/htdocs/LocalSettings.php ${docrootDir}/
rsync -av --progress ${vhost_backup_path}/htdocs/images ${docrootDir}/

# backup LocalSettings.php
cp ${vhostDir}/LocalSettings.php ${vhostDir}/LocalSettings.$(date "+%Y%m%d_%H%M%S").php

# TODO: idempotent commands to update LocalSettings.php

# fix OATHAuth extension failure that breaks upgrade (Error 1146: Table "wiki_oathauth_types" doesn't exist)
#  * https://www.mediawiki.org/wiki/Topic:T24w7s9asgty7vxp
cat $docrootDir/extensions/OATHAuth/sql/sqlite/tables-generated.sql | sed 's/AUTOINCREMENT/AUTO_INCREMENT/g' | sed 's%/\*_\*/%wiki_%g' | mysql -u $dbUser -p${dbPass} $dbName

# FIX PERMISSIONS
time /usr/local/bin/fix_web_permissions.sh

# RUN UPGRADE
time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.43.log

# clear varnish cache (before verify step below)
varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'

# verify by loading:
#
# * https://wiki.opensourceecology.org/wiki/Special:Version
#
# make sure the MediaWiki version speciried is v1.43.0

# UPDATE SKINS

rsync -av --progress /var/tmp/mediawiki/themes/CologneBlue ${docrootDir}/skins/
rsync -av --progress /var/tmp/mediawiki/themes/Modern ${docrootDir}/skins/

# UPDATE EXTENSIONS

extensions="ConfirmAccount UserMerge Widgets DeleteBatch SmiteSpam"

for extension in ${extensions}; do
        extension_path="${docrootDir}/extensions/${extension}"
        source_path="/var/tmp/mediawiki/extensions/${extension}"
        
        if [ -d "${source_path}" ]; then
                echo "${extension}"
                rsync -a ${source_path}/ "${extension_path}/"
        fi
done

# enable new extensions: SmiteSpam & DeleteBatch
grep 'SmiteSpam' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%wfLoadExtension\( 'Nuke' \);%wfLoadExtension( 'Nuke' );\nwfLoadExtension( 'SmiteSpam' );\nwfLoadExtension( 'DeleteBatch' );%igs" ${vhostDir}/LocalSettings.php

# add skins (with wfLoadSkin() instead of require_once)
grep 'wfSkinLoad' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s/$wgDefaultSkin = 'Vector';/\$wgDefaultSkin = 'Vector';\n\nwfLoadSkin( 'CologneBlue' );\nwfLoadSkin( 'Modern' );\nwfLoadSkin( 'MonoBook' );\nwfLoadSkin( 'Vector' );/igs" ${vhostDir}/LocalSettings.php

# re-enable CategoryTree extension
grep 'wfLoadExtension.*CategoryTree' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once\("{\$IP}/extensions/CategoryTree/CategoryTree.php"\);%#require_once("{$IP}/extensions/CategoryTree/CategoryTree.php");\n\nwfLoadExtension( "CategoryTree" );\n%igs' ${vhostDir}/LocalSettings.php

# re-enable ConfirmEdit and switch to FancyCaption
grep 'FancyCaptcha' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%\\\$wgCaptchaClass = 'SimpleCaptcha';%\#\\\$wgCaptchaClass = 'SimpleCaptcha';\n\nwfLoadExtensions([ 'ConfirmEdit', 'ConfirmEdit/FancyCaptcha' ]);\n\\\$wgCaptchaDirectory = '/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/';\n\\\$wgCaptchaSecret = '$wgCaptchaSecret';\n%igs" ${vhostDir}/LocalSettings.php

# update cron script to use new secret
cp /usr/local/bin/mediawiki_generate_captchas.sh /usr/local/bin/mediawiki_generate_captchas.sh.$(date "+%Y%m%d_%H%M%S")
sed -ie "s%CAPTCHA_SECRET='\([^']*\)'%\CAPTCHA_SECRET='$wgCaptchaSecret'%" /usr/local/bin/mediawiki_generate_captchas.sh

# re-enable ConfirmAccount extension
grep 'wfLoadExtension.*ConfirmAccount' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once "\$IP/extensions/ConfirmAccount/ConfirmAccount.php";%#require_once "$IP/extensions/ConfirmAccount/ConfirmAccount.php";\n\nwfLoadExtension( "ConfirmAccount" );\n%igs' ${vhostDir}/LocalSettings.php

# re-enable UserMerge extension
sed -i 's%^\(\s*\)#wfLoadExtension\(.*\)UserMerge\(.*\)%\1wfLoadExtension\2UserMerge\3%' ${vhostDir}/LocalSettings.php

# re-enable Widgets extension
grep 'wfLoadExtension.*Widgets' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once\("\$IP/extensions/Widgets/Widgets.php"\);%#require_once("$IP/extensions/Widgets/Widgets.php");\n\nwfLoadExtension( "Widgets" );\n%igs' ${vhostDir}/LocalSettings.php

# re-enable ParserFunctions extension
grep 'wfLoadExtension.*ParserFunctions' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once\( "\$IP/extensions/ParserFunctions/ParserFunctions.php"  \);%#require_once( "$IP/extensions/ParserFunctions/ParserFunctions.php"  );\n\nwfLoadExtension( "ParserFunctions" );\n%igs' ${vhostDir}/LocalSettings.php

# remove deprecated password policy and add new https://www.mediawiki.org/wiki/Manual:$wgPasswordPolicy
grep 'PasswordNotInCommonList' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%\\\$wgPasswordPolicy\['policies'\]\['default'\]\['PasswordCannotBePopular'\] = PHP_INT_MAX;\n\\\$wgPasswordPolicy\['policies'\]\['default'\]\['PasswordCannotMatchUsername'\] = true;%#\\\$wgPasswordPolicy['policies']['default']['PasswordCannotBePopular'] = PHP_INT_MAX;\n#\\\$wgPasswordPolicy['policies']['default']['PasswordCannotMatchUsername'] = true;\n\\\$wgPasswordPolicy['policies']['default']['PasswordNotInCommonList'] = true;\n\\\$wgPasswordPolicy['policies']['default']['PasswordCannotBeSubstringInUsername'] = true;%igs" ${vhostDir}/LocalSettings.php

# configure OATH (2FA) requirements
grep 'wgOATHAuthWindowRadius' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%wfLoadExtension\( 'OATHAuth' \);%wfLoadExtension( 'OATHAuth' );\n\n# Relaxed mode with +/- 4 minutes time drift tolerance. This is important or users will complain and/or disable 2FA entirely\n# The security consequences are small. The usability benefits are huge.\n\\\$wgOATHAuthWindowRadius = 8;\n\n# make admins require 2FA\n\\\$wgOATHRequiredForGroups = ['sysop', 'interface-admin', 'bureaucrat', 'suppress', 'widgeteditor' ];\n\n# make 'powerful' rights require 2FA\n\\\$wgOATHExclusiveRights =  [ 'apihighlimits', 'applychangetags', 'autoconfirmed', 'autopatrol', 'bigdelete', 'block', 'blockemail', 'bot', 'changetags', 'createaccount', 'createtalk', 'delete', 'deletechangetags', 'deletedhistory', 'deletedtext', 'deletelogentry', 'deleterevision', 'editcontentmodel', 'editinterface', 'editprotected', 'editsemiprotected', 'editsitecss', 'editsitejs', 'editsitejson', 'editusercss', 'edituserjs', 'edituserjson', 'hideuser', 'import', 'importupload', 'ipblock-exempt', 'managechangetags', 'markbotedits', 'mergehistory', 'move-categorypages', 'move-rootuserpages', 'move-subpages', 'nominornewtalk', 'noratelimit', 'patrol', 'protect', 'renameuser', 'reupload-shared', 'sendemail', 'suppressionlog', 'suppressredirect', 'suppressrevision', 'unblockself', 'unwatchedpages', 'userrights', ];\n\n# full list for reference (taken from htdocs/includes/MainConfigSchema.php) https://www.mediawiki.org/wiki/Extension:OATHAuth#Configuration\n#\\\$wgOATHExclusiveRights =  [ 'apihighlimits', 'applychangetags', 'autoconfirmed', 'autopatrol', 'bigdelete', 'block', 'blockemail', 'bot', 'browsearchive', 'changetags', 'createaccount', 'createpage', 'createtalk', 'delete', 'deletechangetags', 'deletedhistory', 'deletedtext', 'deletelogentry', 'deleterevision', 'edit', 'editcontentmodel', 'editinterface', 'editmyoptions', 'editmyprivateinfo', 'editmyusercss', 'editmyuserjs', 'editmyuserjson', 'editmyuserjsredirect', 'editmywatchlist', 'editprotected', 'editsemiprotected', 'editsitecss', 'editsitejs', 'editsitejson', 'editusercss', 'edituserjs', 'edituserjson', 'hideuser', 'import', 'importupload', 'ipblock-exempt', 'managechangetags', 'markbotedits', 'mergehistory', 'minoredit', 'move', 'move-categorypages', 'movefile', 'move-rootuserpages', 'move-subpages', 'nominornewtalk', 'noratelimit', 'patrol', 'protect', 'read', 'renameuser', 'reupload', 'reupload-shared', 'rollback', 'sendemail', 'suppressionlog', 'suppressredirect', 'suppressrevision', 'unblockself', 'undelete', 'unwatchedpages', 'upload', 'userrights', 'viewmyprivateinfo', 'viewmywatchlist', 'viewsuppressed' ];\n%igs" ${vhostDir}/LocalSettings.php

# FIX PERMISSIONS
time /usr/local/bin/fix_web_permissions.sh

# clear varnish cache again
varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'

# log into the new MediaWiki site, and update settings:

# Create a new account (Called 'CatarinaAdmin') and add it to sysop group

# Remove the following accounts from the sysop group
## Audrey Rampone
## Elifarley
## Hart
## Will

# TODO: Enforce 2FA for all admin accounts

TODO: Update DNS to point to hetzner3

1. wait until DNS changes to hetzner3

while true; do date; dig store.opensourceecology.org; sleep 10; echo; done

Validation Steps

1. TODO: copy from previous CHGs

Revert Steps

TODO

See Also

1. CHG-2018-02-05_migrate_osemain_to_hetzner2 Last wordpress migration from hetzner1 to hetzner2
2. CHG-2024-07-26 yum update Last (possible) update to hetzner2
3. List of other CHG "tickets"