CHG-2024-07-26 yum update

From Open Source Ecology
Jump to navigation Jump to search

Status

2024-07-24 05:30 UTC

Creating first draft of this change ticket

Change Info

Scheduled Time

This change will take place on 2024-07-26 16:00 UTC

= 2024-07-26 11:00 Kansas City, US = 2024-07-26 11:00 Guayaquil, EC

https://www.timeanddate.com/worldclock/converter.html?iso=20240727T160000&p1=405&p2=1440&p3=93

Points of Contact

Change being performed by: Michael Altfield

Service owners: Catarina Mota & Marcin Jakubowski

Purpose

This CHG will update all the yum-installed packages on the server.

The last system-wide `yum update` appears to have been preformed on 2020-05-04, so we're well over-due for an update.

Note that security-critical updates are already being updated automatically via `yum-cron`.

Time Length

This whole process is expected to take 1-3 hours.

Some systems could be impacted for days, if issues are encountered.

Systems Impacted

The production server will be restarted after the update to apply kernel changes, and this will necessarily bring all of OSE's sites offline temporarily.

Staging Test

TODO

Pre-state Proof

The below command will output a ton of packages requiring updates before this CHG 

yum list updates

Change Steps

# open screen
screen -S CHG-2024-07-26_yum_update

# become root
sudo su -

# confirm that backups have finished uploading to B2
less /var/log/backups/backup.log
sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups | grep `date "+%Y%m%d"`
date -u
sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups

# create dir for logging the change
tmpDir="/var/tmp/CHG-2024-07-26_yum_update"
mkdir -p $tmpDir

# begin to gracefully shutdown nginx in the background
time nice /sbin/nginx -s quit

# first record the set of packages currently installed
time nice rpm -qa &> "${tmpDir}/before.log"

# update packages
time nice yum -y update-to TODO &> "${tmpDir}/update.log"

# log the post-state packages and versions
time nice rpm -qa &> "${tmpDir}/after.log"

# check to see what changes require a reboot, if any
time nice needs-restarting &> "${tmpDir}/needs-restarting.log"
time nice needs-restarting -r &> "${tmpDir}/needs-reboot.log"

# reboot the system to apply kernel changes
reboot

# after reboot, initiate a new backup
time sudo /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

Post-state Proof

The below command will output zero or nearly-zero packages after this CHG is successfully complete 

yum list updates

Validation Steps

  1. Access the following URLs; make sure redirects and the destination pages look sane
    1. http://wiki.opensourceecology.org/
    2. http://opensourceecology.org/
    3. http://openbuildinginstitute.org/
    4. http://store.opensourceecology.org/
    5. http://oswh.opensourceecology.org/
    6. http://fef.opensourceecology.org/
    7. http://microfactory.opensourceecology.org/
    8. http://forum.opensourceecology.org/
    9. http://phplist.opensourceecology.org/lists/
  1. Access and login to the following private sites too to make sure everything is sane
    1. https://awstats.opensourceecology.org:4443
    2. https://munin.opensourceecology.org:4443
  1. Login and attempt to make a trivial change on the following sites
    1. https://wiki.opensourceecology.org/
    2. https://opensourceecology.org/

Revert Steps

All the packages can be reverted to their previous versions using the following command 

yum update-to TODO

See Also

  1. https://serverfault.com/questions/1014455/how-to-list-packages-needing-update-in-format-for-yum-update-to
  2. CHG-2020-05-04_yum_update Last yum update (in May 2020)
  3. List of other CHG "tickets"
Retrieved from "https://wiki.opensourceecology.org/index.php?title=CHG-2024-07-26_yum_update&oldid=298903"