Discourse/Updating
This guide describes how to update Discourse installed on our production OSE_Server.
For a guide on how to install Discourse on CentOS 7, see Discourse/Install.
To see our general Discourse page, see Discourse.
Updating Discoruse
This section will describe how to update the Discourse software
Discourse Versions
First, a note about Discourse releases: Discourse maintains a "stable" release, but they don't actually backport bug patches to their stable releases like one would expect. There's no LTS (or STS!) for Discourse stable releases.
They only fix bugs in future beta releases (which will also include new commits that may break more things). Therefore, the default branch for production is "beta" releases, and they urge their customers not to use "stable." I honestly think this is a terrible idea, especially for a small org like OSE without any full-time ops staff to constantly update our prod apps.
* https://github.com/discourse/discourse/releases
IMHO, the takeaway to this is that updates should be done to Discourse very carefully and with through testing in staging before taking a backup and following the identical procedure on production.
Check for Updates
Discourse updates often, and it will tell you to update the app (requiring downtime) even if there's 1 (untested?) commit to their discourse github repository
First, determine your current version by logging-in and visiting the Admin section. You'll see the version listed as a version number (ex: 2.5.0.beta3) and a git commit hash (ex: 8e28ccb2ea). Be sure to write down the current version in your notes in-case something goes wrong during the upgrade.
* https://discourse.opensourceecology.org/admin
And you can determine the latest version by checking the Discourse projects' github releases section
* https://github.com/discourse/discourse/releases
Step 0: Trigger Backup Scripts for System-Wide backup
For good measure, trigger a backup of the entire system's database & files:
sudo su - time sudo /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log exit
When finished, list the backup files in our Backblaze B2 server backups bucket to verify that the whole system backup was successful before proceeding
source /root/backups/backup.settings
$SUDO -u ${b2UserName} $B2 ls ${B2_BUCKET_NAME}
Step 1: Set variables
Type these commands to set some variables, which will be used by the commands in the sections below. Replace 'www.opensourceecology.org' with the corresponding directory for the wp site you're updating.
sudo su -
export vhostDir="/var/discourse/"
export stamp=`date +%Y%m%d_%T`
export tmpDir="/var/tmp/discourseUpgrade.${stamp}"
# verify
echo "${vhostDir}"
echo "${stamp}"
echo "${tmpDir}"
ls -lah "${vhostDir}"
Step 2: Make Vhost-specific backups
The backups made in the previous step are huge. Because it's easier to work with vhost-specific backups, let's make a redundant copy available in /var/tmp/:
mkdir "${tmpDir}"
chown root:root "${tmpDir}"
chmod 0700 "${tmpDir}"
pushd "${tmpDir}"
# discourse backup (db & uploaded files only)
nice rm -rf /var/discourse/shared/standalone/backups/default/*.tar.gz
time nice docker exec discourse_ose discourse backup
nice mv /var/discourse/shared/standalone/backups/default/*.tar.gz ${tmpDir}/
# files backup (all discourse files)
time nice tar --exclude "${vhostDir}/shared/standalone/postgres_data" --exclude "${vhostDir}/shared/standalone/postgres_data/uploads" --exclude "${vhostDir}/shared/standalone/backups" -czf ${tmpDir}/discourse_files.${stamp}.tar.gz /var/discourse/*
popd
Step 3: Update git
First, try a `git pull origin master` from the '/var/discourse' directory. If it says that you're "Already up-to-date.", then skip this section. If you have merge errors, then you'll need to proceed with this section.
We've made some OSE-specific changes to the files in /var/discourse that conflict with the upstream git repo, so let's move those out of the way before updating. After the git pull, we'll update them again.
First execute the commands shown in the example below as root to get a list of all the files that have been modified locally.
[root@osestaging1 /]# pushd /var/discourse /var/discourse / [root@osestaging1 discourse]# git status --short | grep -E '^\s*M' M image/base/install-nginx M launcher [root@osestaging1 discourse]# popd / [root@osestaging1 /]#
In the example above, we see that the '/var/discourse/launcher' script and '/var/discourse/image/base/install-nginx' script have both been modified in our custom OSE install. Now let's move those two files out of the way so we can do a clean `git pull` after.
Execute the following commands as root. If there were more than just the two files listed above that were modified, then you should update this documentation to reflect the necessary changes.
export vhostDir="/var/discourse/"
stamp=`date "+%Y%m%d_%H%M%S"`
# launcher
mv "${vhostDir}/launcher" "${vhostDir}/launcher.${stamp}"
git checkout "${vhostDir}/launcher"
# install-nginx
mv "${vhostDir}/image/base/install-nginx" "${vhostDir}/image/base/install-nginx.${stamp}"
git checkout "${vhostDir}/image/base/install-nginx"
Now sync to the upstream repo
pushd "${vhostDir}"
git pull origin master
popd
Before proceeding, let's get a diff of the changes so we can understand what may have changed upstream, which might break our commands in our OSE Discourse install guide
diff "${vhostDir}/image/base/install-nginx.${stamp}" "${vhostDir}/image/base/install-nginx"
diff "${vhostDir}/launcher.${stamp}" "${vhostDir}/launcher"
Write down the output from the above commands in your log; you may need it to debug later.
Next, to re-apply the changes to the 'install-nginx' script and "launcher' script, see the following relevant section for a list of the necessary commands:
See also https://meta.discourse.org/t/how-do-i-manually-update-discourse-and-docker-image-to-latest/23325
Step 4: Build Discourse docker image with OSE modifications
Now we must rebuild the upstream Discourse Docker image with our OSE-specific modifications, such as giving nginx mod_security support.
pushd "${vhostDir}/image/base"
# force a fresh build (no-cache) so the `git pull` lines will trigger
# note this will take a *ridiculously* long time; the Discourse team compiles many packages from source :(
time nice docker build --no-cache --network=host --tag 'discourse_ose' /var/discourse/image/base/
popd
Step 5: Rebuild the app
time nice ${vhostDir}/launcher rebuild discourse_ose
You may also need to restart the "outer" nginx on the docker host too
systemctl restart nginx
Creating a Backup
To create a a backup of Discourse, see the "Updating Discourse" Steps 0-2 starting with #Step 0: Trigger Backup Scripts for System-Wide backup
Restoring from Backup
This section will describe how to restore Discourse from a previous backup.
Discourse backup file
The best way to restore Discourse from backup is to use the "proper" backup file that was safely generated by the Discourse app itself. After retreiving the backup file, copy it to the '/var/discourse/shared/standalone/backups/default' directory
[root@osestaging1 discourse_ose]# [ -f /var/discourse/shared/standalone/backups/default/ ] || mkdir /var/discourse/shared/standalone/backups/default/ [root@osestaging1 discourse_ose]# cp discourse-2020-03-08-172140-v20191219112000.tar.gz /var/discourse/shared/standalone/backups/default/ [root@osestaging1 discourse_ose]# ls -lah /var/discourse/shared/standalone/backups/default/ total 56M drwxr-xr-x. 2 root root 4.0K Mar 16 16:52 . drwxr-xr-x. 3 tgriffing 33 4.0K Mar 16 16:52 .. -rw-r--r--. 1 root root 56M Mar 16 16:52 discourse-2020-03-08-172140-v20191219112000.tar.gz [root@osestaging1 discourse_ose]#
Then execute the following commands as root to initiate the restore
[root@osestaging1 discourse]# /var/discourse/launcher enter discourse_ose root@osestaging1-discourse-ose:/var/www/discourse# discourse enable_restore Restore are now permitted. Disable them with `disable_restore` root@osestaging1-discourse-ose:/var/www/discourse# discourse restore discourse-2020-03-08-172140-v20191219112000.tar.gz Starting restore: discourse-2020-03-08-172140-v20191219112000.tar.gz [STARTED] 'system' has started the restore! Marking restore as running... Making sure /var/www/discourse/tmp/restores/default/2020-03-16-165545 exists... Copying archive to tmp directory... Unzipping archive, this may take a while... Extracting dump file... Validating metadata... Current version: 20200311135425 Restored version: 20191219112000 Enabling readonly mode... Pausing sidekiq... Waiting up to 60 seconds for Sidekiq to finish running jobs... Creating missing functions in the discourse_functions schema... Restoring dump file... (this may take a while) ... Cleaning stuff up... Dropping functions from the discourse_functions schema... Removing tmp '/var/www/discourse/tmp/restores/default/2020-03-16-165545' directory... Unpausing sidekiq... Marking restore as finished... Notifying 'system' of the end of the restore... Finished! [SUCCESS] Restore done. root@osestaging1-discourse-ose:/var/www/discourse#
That's it! The Discourse site should now be fully restored. After verification, be sure to disable the restore function again
root@osestaging1-discourse-ose:/var/www/discourse# discourse disable_restore Restore are now forbidden. Enable them with `enable_restore` root@osestaging1-discourse-ose:/var/www/discourse#
Note: You may need to restart the docker container to complete the restore:
/var/discourse/launcher stop discourse_ose /var/discourse/launcher start discourse_ose
File backup file
Our backup cron also takes a tar of most of the '/var/discourse' directory, but this process doesn't stop the Discourse application, so it's an option for restore as a last-resort and not covered here.
