CHG-2025-XX-XX deprecate oswh - Revision history

Maltfield: s/http/https/

2025-01-30T20:48:43Z

s/http/https/

← Older revision		Revision as of 20:48, 30 January 2025
Line 195:		Line 195:
	find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/www\.opensourcewarehouse\.org/oswh.opensourceecology.org/g' '{}' \;		find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/www\.opensourcewarehouse\.org/oswh.opensourceecology.org/g' '{}' \;
	find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse\.org/oswh.opensourceecology.org/g' '{}' \;		find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse\.org/oswh.opensourceecology.org/g' '{}' \;

			# and fix some http to become https
			find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://www.youtube.com%https://www.youtube.com%g' '{}' \;
			find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://player.vimeo.com%https://player.vimeo.com%g' '{}' \;
			find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://oswh.opensourceecology.org%https://oswh.opensourceecology.org%g' '{}' \;

	# SET PERMISSIONS		# SET PERMISSIONS

Maltfield: fix missing js assets

2025-01-30T20:35:42Z

fix missing js assets

← Older revision		Revision as of 20:35, 30 January 2025
Line 171:		Line 171:
	EOF		EOF

	# STEP 5: Add all uploaded files (fix missing from wget)		# STEP 5: Add missing files

			# all uploaded files (fix missing from wget)
	tar -xjvf ${backupFileName_files_hetzner2}		tar -xjvf ${backupFileName_files_hetzner2}
	rsync -av --progress var/www/html/${vhost_name}/htdocs/wp-content/uploads/ ${vhostDir}/htdocs/wp-content/uploads/		rsync -av --progress var/www/html/${vhost_name}/htdocs/wp-content/uploads/ ${vhostDir}/htdocs/wp-content/uploads/

			# javascript files
			rsync -av --progress --include=".js" --include="/" --exclude="*" var/www/html/${vhost_name}/htdocs/wp-content/themes/Eventor/ ${vhostDir}/htdocs/wp-content/themes/Eventor/

	# STEP 6: Fix assets		# STEP 6: Fix assets

Maltfield: www.oswh.opensourceecology.org isn't going to work

2025-01-30T04:17:21Z

www.oswh.opensourceecology.org isn't going to work

← Older revision		Revision as of 04:17, 30 January 2025
Line 188:		Line 188:

	# also replace all occurrences of 'openwarehouse.org' with 'oswh.opensourceecology.org'		# also replace all occurrences of 'openwarehouse.org' with 'oswh.opensourceecology.org'
	find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse.org/oswh.opensourceecology.org/g' '{}' \;		find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/www\.opensourcewarehouse\.org/oswh.opensourceecology.org/g' '{}' \;
			find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse\.org/oswh.opensourceecology.org/g' '{}' \;

	# SET PERMISSIONS		# SET PERMISSIONS

Maltfield: gotta replace it everwhere

2025-01-30T04:07:44Z

gotta replace it everwhere

← Older revision		Revision as of 04:07, 30 January 2025
Line 188:		Line 188:

	# also replace all occurrences of 'openwarehouse.org' with 'oswh.opensourceecology.org'		# also replace all occurrences of 'openwarehouse.org' with 'oswh.opensourceecology.org'
	find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse.org/oswh.opensourceecology.org/' '{}' \;		find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse.org/oswh.opensourceecology.org/g' '{}' \;

	# SET PERMISSIONS		# SET PERMISSIONS

Maltfield at 04:04, 30 January 2025

2025-01-30T04:04:03Z

← Older revision		Revision as of 04:04, 30 January 2025
Line 284:		Line 284:
	# [[CHG-2018-02-05_migrate_osemain_to_hetzner2]] Last wordpress migration from hetzner1 to hetzner2		# [[CHG-2018-02-05_migrate_osemain_to_hetzner2]] Last wordpress migration from hetzner1 to hetzner2
	# [[CHG-2024-07-26 yum update]] Last (possible) update to hetzner2		# [[CHG-2024-07-26 yum update]] Last (possible) update to hetzner2
			# [[CHG-2025-XX-XX_deprecate_fef]]
	# [[:Category: CHGs\|List of other CHG "tickets"]]		# [[:Category: CHGs\|List of other CHG "tickets"]]
	# [[CHG-2018-02-04_deprecate_vanilla_forums]]		# [[CHG-2018-02-04_deprecate_vanilla_forums]]

	[[Category: CHGs]]		[[Category: CHGs]]

Maltfield: initial draft, created from fef page

2025-01-30T04:03:39Z

initial draft, created from fef page

New page

=Status=

==2025-01-30 04:01 UTC==

Initial Ticket draft created on wiki (WIP)

=Change Info=

==Scheduled Time==

This change will take place on 2025-??-?? ??:00 UTC

* = 2025-??-?? ??:00 Kansas City, US
* = 2025-??-?? ??:00 Guayaquil, EC

https://www.timeanddate.com/worldclock/converter.html?iso=20240727T160000&p1=405&p2=1440&p3=93

=Purpose=

This change will do the following for the oswh site:

# entirely migrate the OSE oswh site from [[hetzner2]] (EOL CentOS7) to [[hetzner3]] (Debian 12)
# changes site from apache v2.4.6 to v2.4.62
# changes site from varnish v4.0.5 to v7.1.1
# changes site from nginx v1.20.1 to v1.22.1
# deprecates the site without loosing the content. we're replacing our oswh site (now with an extremely out-of-date & broken wordpress theme) with purely static content made from a wget scrape
# keeps a backup of the dynamic codebase & database on the server, but 1 directory _above_ the docroot.

=Points of Contact=

Change being performed by: [[User:Maltfield|Michael Altfield]]

Service owners: [[User:Catarina|Catarina Mota]] & [[User:Marcin|Marcin Jakubowski]]

==Time Length==

This whole process is expected to take 6-24 hours.

Some systems could be impacted for days, if issues are encountered.

==Systems Impacted==

This change impacts both hetzner2 and hetzner3. It's possible that we'll need to restart one or both of these servers during this migration, which could affect the availability of all OSE & OBI websites.

==Staging Test==

n/a

=Change Steps=
<pre>
####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
source /root/backups/backup.settings
/root/backups/backup.sh

# when finished, verify that the whole system backup was successful before proceeding
sudo rclone lsl b2:ose-server-backups | grep -i $(date +%Y%m%d)

vhost_name='oswh.opensourceecology.org'
dbName='oswh_db'
dbUser="CHANGEME"
dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.bz2"
vhostDir="/var/www/html/${vhost_name}"

# STEP 1: Bring site down on hetzner2
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current

# backup nginx config
cp /etc/nginx/conf.d/${vhost_name}.conf nginx_${vhost_name}.${stamp}.conf

# restrict website to local requests only
grep 'deny' /etc/nginx/conf.d/${vhost_name}.conf || sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%^$\s*$server_name$.*$%\1server_name\2\n\tallow 127.0.0.1;\n\tallow 2a01:4f8:172:209e::2;\n\tallow 138.201.84.243;\n\tdeny all;\n%' /etc/nginx/conf.d/${vhost_name}.conf

nginx -t && service nginx reload

# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/

time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}

# STEP 3: BACKUP FILES
time nice tar -cjvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}

# STEP 4: GENERATE STATIC SITE
mkdir wget
pushd wget
time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}"

# STEP 5: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/

####################
# run on hetzner3 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, verify that the whole system backup was successful before proceeding
rclone lsl b2:ose-server-backups | grep -i $(date +%Y%m%d)

# DECLARE VARIABLES
vhost_name='oswh.opensourceecology.org'
dbName='oswh_db'
dbUser="CHANGEME"
dbPass="CHANGEME"

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.bz2"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"

# STEP 1: ADD DB

# first, get rid of the old vhost dir
mv "${vhostDir}" "${backupDir_hetzner3}/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
mkdir -p "${vhostDir}"

# put a copy of the db file in the vhost dir (outside docroot)
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${vhostDir}/
chown root:root ${vhostDir}/*.sql.bz2
chmod 0400 ${vhostDir}/*.sql.bz2

time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};"

# STEP 2: Add vhost files backup to vhost dir (outside docroot)
cp ${backupFileName_files_hetzner2} ${vhostDir}/
chown root:root ${vhostDir}/*.tar.bz2
chmod 0400 ${vhostDir}/*.tar.bz2

# STEP 3: Add static html files to docroot
time nice rsync -av --progress wget/${vhost_name}/ ${docrootDir}/

# STEP 4: Create README about site deprecation and how to restore it
cat << EOF > ${vhostDir}/README.txt
In 2024, this website was no longer updated, the code required to render it was broken, and the decision was made to deprecate support for the site. The content is still accessible in as static-content; new content is not possible.

For more information, please see:

* https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_deprecate_fef

In 2025, during the backup stage of the change to deprecate this website, a backup of the wordpress webroot directory and database dump were created for upload to long-term backup storage on hetzner3. Because this backup size was manageable small, I put a "hot" copy of this dump in this website's apache vhost dir (but outside the htdocs docroot, of course).

-- Michael Altfield <michael@michaelaltfield.net> $(date -u --rfc-3339=seconds)

EOF

# STEP 5: Add all uploaded files (fix missing from wget)
tar -xjvf ${backupFileName_files_hetzner2}
rsync -av --progress var/www/html/${vhost_name}/htdocs/wp-content/uploads/ ${vhostDir}/htdocs/wp-content/uploads/

# STEP 6: Fix assets

# wget html-encoded question marks in versioned assets, which causes "MIME type mismatch" errors; change it back
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/\.js%3F/.js?/' '{}' \;
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/\.css%3F/.css?/' '{}' \;

# also rename the assets as back to their original filenames
js_files=$(find ${vhostDir}/htdocs/ -type f -iname '*\.js\?*')
for f in $js_files; do mv $f $(echo $f | sed 's/\.js\?.*/.js/'); done
css_files=$(find ${vhostDir}/htdocs/ -type f -iname '*\.css\?*')
for f in $css_files; do mv $f $(echo $f | sed 's/\.css\?.*/.css/'); done

# also replace all occurrences of 'openwarehouse.org' with 'oswh.opensourceecology.org'
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse.org/oswh.opensourceecology.org/' '{}' \;

# SET PERMISSIONS

# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;

#############
# WORDPRESS #
#############

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do

wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"

chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;

chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"

[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;

[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;

done

###########
# phpList #
###########

phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"

for vhost_dir in $phplist_sites; do

for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done

for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done

done

#############
# MediaWiki #
#############

vhost_dir="/var/www/html/wiki.opensourceecology.org"
mw_docroot="${vhost_dir}/htdocs"

chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;

chown not-apache:apache-admins "${vhost_dir}/LocalSettings.php"
chmod 0040 "${vhost_dir}/LocalSettings.php"

[ -d "${mw_docroot}/images" ] || mkdir "${mw_docroot}/images"
chown -R www-data:www-data "${mw_docroot}/images"
find "${mw_docroot}/images" -type f -exec chmod 0660 {} \;
find "${mw_docroot}/images" -type d -exec chmod 0770 {} \;

[ -d "${vhost_dir}/cache" ] || mkdir "${vhost_dir}/cache"
chown -R www-data:www-data "${vhost_dir}/cache"
find "${vhost_dir}/cache" -type f -exec chmod 0660 {} \;
find "${vhost_dir}/cache" -type d -exec chmod 0770 {} \;

</pre>

==Validation Steps==

# TODO: copy from previous CHGs

==Revert Steps==

TODO

=See Also=

# [[CHG-2018-02-05_migrate_osemain_to_hetzner2]] Last wordpress migration from hetzner1 to hetzner2
# [[CHG-2024-07-26 yum update]] Last (possible) update to hetzner2
# [[:Category: CHGs|List of other CHG "tickets"]]
# [[CHG-2018-02-04_deprecate_vanilla_forums]]

[[Category: CHGs]]