From Open Source Ecology
Jump to: navigation, search

This guide describes how to update Discourse installed on our production OSE_Server.

For a guide on how to install Discourse on CentOS 7, see Discourse/Install.

To see our general Discourse page, see Discourse.

Updating Discoruse

This section will describe how to update the Discourse software

Discourse Versions

First, a note about Discourse releases: Discourse maintains a "stable" release, but they don't actually backport bug patches to their stable releases like one would expect. There's no LTS (or STS!) for Discourse stable releases.

They only fix bugs in future beta releases (which will also include new commits that may break more things). Therefore, the default branch for production is "beta" releases, and they urge their customers not to use "stable." I honestly think this is a terrible idea, especially for a small org like OSE without any full-time ops staff to constantly update our prod apps.

* https://github.com/discourse/discourse/releases

IMHO, the takeaway to this is that updates should be done to Discourse very carefully and with through testing in staging before taking a backup and following the identical procedure on production.

Check for Updates

Discourse updates often, and it will tell you to update the app (requiring downtime) even if there's 1 (untested?) commit to their discourse github repository

First, determine your current version by logging-in and visiting the Admin section. You'll see the version listed as a version number (ex: 2.5.0.beta3) and a git commit hash (ex: 8e28ccb2ea). Be sure to write down the current version in your notes in-case something goes wrong during the upgrade.

* https://discourse.opensourceecology.org/admin

And you can determine the latest version by checking the Discourse projects' github releases section

* https://github.com/discourse/discourse/releases

Step 0: Trigger Backup Scripts for System-Wide backup

For good measure, trigger a backup of the entire system's database & files:

sudo su -
time sudo /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

When finished, list the backup files in our Backblaze B2 server backups bucket to verify that the whole system backup was successful before proceeding

source /root/backups/backup.settings
$SUDO -u ${b2UserName} $B2 ls ${B2_BUCKET_NAME}

Step 1: Set variables

Type these commands to set some variables, which will be used by the commands in the sections below. Replace 'www.opensourceecology.org' with the corresponding directory for the wp site you're updating.

sudo su -
export vhostDir="/var/discourse/"
export stamp=`date +%Y%m%d_%T`
export tmpDir="/var/tmp/discourseUpgrade.${stamp}"

# verify
echo "${vhostDir}"
echo "${stamp}"
echo "${tmpDir}"
ls -lah "${vhostDir}"

Step 2: Make Vhost-specific backups

The backups made in the previous step are huge. Because it's easier to work with vhost-specific backups, let's make a redundant copy available in /var/tmp/:

mkdir "${tmpDir}"
chown root:root "${tmpDir}"
chmod 0700 "${tmpDir}"
pushd "${tmpDir}"

# discourse backup (db & uploaded files only)
nice rm -rf /var/discourse/shared/standalone/backups/default/*.tar.gz
time nice docker exec discourse_ose discourse backup
nice mv /var/discourse/shared/standalone/backups/default/*.tar.gz ${tmpDir}/

# files backup (all discourse files)
time nice tar --exclude "${vhostDir}/shared/standalone/postgres_data" --exclude "${vhostDir}/shared/standalone/postgres_data/uploads" --exclude "${vhostDir}/shared/standalone/backups" -czf ${tmpDir}/discourse_files.${stamp}.tar.gz /var/discourse/*


Step 3: Update git

First, try a `git pull origin master` from the '/var/discourse' directory. If it says that you're "Already up-to-date.", then skip this section. If you have merge errors, then you'll need to proceed with this section.

We've made some OSE-specific changes to the files in /var/discourse that conflict with the upstream git repo, so let's move those out of the way before updating. After the git pull, we'll update them again.

First execute the commands shown in the example below as root to get a list of all the files that have been modified locally.

[root@osestaging1 /]# pushd /var/discourse
/var/discourse /
[root@osestaging1 discourse]# git status --short | grep -E '^\s*M'
 M image/base/install-nginx
 M launcher
[root@osestaging1 discourse]# popd
[root@osestaging1 /]# 

In the example above, we see that the '/var/discourse/launcher' script and '/var/discourse/image/base/install-nginx' script have both been modified in our custom OSE install. Now let's move those two files out of the way so we can do a clean `git pull` after.

Execute the following commands as root. If there were more than just the two files listed above that were modified, then you should update this documentation to reflect the necessary changes.

export vhostDir="/var/discourse/"
stamp=`date "+%Y%m%d_%H%M%S"`

# launcher
mv "${vhostDir}/launcher" "${vhostDir}/launcher.${stamp}"
git checkout "${vhostDir}/launcher"

# install-nginx
mv "${vhostDir}/image/base/install-nginx" "${vhostDir}/image/base/install-nginx.${stamp}"
git checkout "${vhostDir}/image/base/install-nginx"

Now sync to the upstream repo

pushd "${vhostDir}"
git pull origin master

Before proceeding, let's get a diff of the changes so we can understand what may have changed upstream, which might break our commands in our OSE Discourse install guide

diff "${vhostDir}/image/base/install-nginx.${stamp}" "${vhostDir}/image/base/install-nginx"
diff "${vhostDir}/launcher.${stamp}" "${vhostDir}/launcher"

Write down the output from the above commands in your log; you may need it to debug later.

Next, to re-apply the changes to the 'install-nginx' script and "launcher' script, see the following relevant section for a list of the necessary commands:

  1. #Nginx_mod_security

See also https://meta.discourse.org/t/how-do-i-manually-update-discourse-and-docker-image-to-latest/23325

Step 4: Build Discourse docker image with OSE modifications

Now we must rebuild the upstream Discourse Docker image with our OSE-specific modifications, such as giving nginx mod_security support.

pushd "${vhostDir}/image/base"

# force a fresh build (no-cache) so the `git pull` lines will trigger
# note this will take a *ridiculously* long time; the Discourse team compiles many packages from source :(
time nice docker build --no-cache --network=host --tag 'discourse_ose' /var/discourse/image/base/


Step 5: Rebuild the app

time nice ${vhostDir}/launcher rebuild discourse_ose

You may also need to restart the "outer" nginx on the docker host too

systemctl restart nginx

Creating a Backup

To create a a backup of Discourse, see the "Updating Discourse" Steps 0-2 starting with #Step 0: Trigger Backup Scripts for System-Wide backup

Restoring from Backup

This section will describe how to restore Discourse from a previous backup.

Discourse backup file

The best way to restore Discourse from backup is to use the "proper" backup file that was safely generated by the Discourse app itself. After retreiving the backup file, copy it to the '/var/discourse/shared/standalone/backups/default' directory

[root@osestaging1 discourse_ose]# [ -f /var/discourse/shared/standalone/backups/default/ ] || mkdir /var/discourse/shared/standalone/backups/default/
[root@osestaging1 discourse_ose]# cp discourse-2020-03-08-172140-v20191219112000.tar.gz /var/discourse/shared/standalone/backups/default/
[root@osestaging1 discourse_ose]# ls -lah /var/discourse/shared/standalone/backups/default/
total 56M
drwxr-xr-x. 2 root      root 4.0K Mar 16 16:52 .
drwxr-xr-x. 3 tgriffing   33 4.0K Mar 16 16:52 ..
-rw-r--r--. 1 root      root  56M Mar 16 16:52 discourse-2020-03-08-172140-v20191219112000.tar.gz
[root@osestaging1 discourse_ose]# 

Then execute the following commands as root to initiate the restore

[root@osestaging1 discourse]# /var/discourse/launcher enter discourse_ose
root@osestaging1-discourse-ose:/var/www/discourse# discourse enable_restore
Restore are now permitted. Disable them with `disable_restore`
root@osestaging1-discourse-ose:/var/www/discourse# discourse restore discourse-2020-03-08-172140-v20191219112000.tar.gz
Starting restore: discourse-2020-03-08-172140-v20191219112000.tar.gz
'system' has started the restore!
Marking restore as running...
Making sure /var/www/discourse/tmp/restores/default/2020-03-16-165545 exists...
Copying archive to tmp directory...
Unzipping archive, this may take a while...
Extracting dump file...
Validating metadata...
  Current version: 20200311135425
  Restored version: 20191219112000
Enabling readonly mode...
Pausing sidekiq...
Waiting up to 60 seconds for Sidekiq to finish running jobs...
Creating missing functions in the discourse_functions schema...
Restoring dump file... (this may take a while)
Cleaning stuff up...
Dropping functions from the discourse_functions schema...
Removing tmp '/var/www/discourse/tmp/restores/default/2020-03-16-165545' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Restore done.

That's it! The Discourse site should now be fully restored. After verification, be sure to disable the restore function again

root@osestaging1-discourse-ose:/var/www/discourse# discourse disable_restore
Restore are now forbidden. Enable them with `enable_restore`

Note: You may need to restart the docker container to complete the restore:

/var/discourse/launcher stop discourse_ose
/var/discourse/launcher start discourse_ose

File backup file

Our backup cron also takes a tar of most of the '/var/discourse' directory, but this process doesn't stop the Discourse application, so it's an option for restore as a last-resort and not covered here.



  1. Discourse
  2. Discourse/Install