CHG-2024-07-26 yum update

From Open Source Ecology
Jump to navigation Jump to search

Status

2024-07-24 05:30 UTC

Creating first draft of this change ticket

Change Info

Scheduled Time

This change will take place on 2024-07-26 16:00 UTC

= 2024-07-26 11:00 Kansas City, US = 2024-07-26 11:00 Guayaquil, EC

https://www.timeanddate.com/worldclock/converter.html?iso=20240727T160000&p1=405&p2=1440&p3=93

Points of Contact

Change being performed by: Michael Altfield

Service owners: Catarina Mota & Marcin Jakubowski

Purpose

This CHG will update all the yum-installed packages on the server.

The last system-wide `yum update` appears to have been preformed on 2020-05-04, so we're well over-due for an update.

Note that security-critical updates are already being updated automatically via `yum-cron`.

Time Length

This whole process is expected to take 1-3 hours.

Some systems could be impacted for days, if issues are encountered.

Systems Impacted

The production server will be restarted after the update to apply kernel changes, and this will necessarily bring all of OSE's sites offline temporarily.

Staging Test

TODO

Pre-state Proof

The below command will output a ton of packages requiring updates before this CHG 

yum list updates

Change Steps

# open screen
screen -S CHG-2024-07-26_yum_update

# become root
sudo su -

# confirm that backups have finished uploading to B2
less /var/log/backups/backup.log
sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups | grep `date "+%Y%m%d"`
date -u
sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups

# create dir for logging the change
tmpDir="/var/tmp/CHG-2024-07-26_yum_update"
mkdir -p $tmpDir

# begin to gracefully shutdown nginx in the background
time nice /sbin/nginx -s quit

# first record the set of packages currently installed
time nice rpm -qa &> "${tmpDir}/before.log"

# update packages
time nice yum -y update-to apr-1.4.8-7.el7 apr-util-1.5.2-6.el7_9.1 augeas-libs-1.4.0-10.el7 bash-4.2.46-35.el7_9 bind-export-libs-9.11.4-26.P2.el7_9.16 bind-libs-9.11.4-26.P2.el7_9.16 bind-libs-lite-9.11.4-26.P2.el7_9.16 bind-license-9.11.4-26.P2.el7_9.16 bind-utils-9.11.4-26.P2.el7_9.16 binutils-2.27-44.base.el7_9.1 ca-certificates-2023.2.60_v7.0.306-72.el7_9 centos-release-7-9.2009.2.el7.centos chkconfig-1.7.6-1.el7 copy-jdk-configs-3.3-11.el7_9 coreutils-8.22-24.el7_9.2 cpio-2.11-28.el7 cpp-4.8.5-44.el7 cronie-1.4.11-25.el7_9 cronie-anacron-1.4.11-25.el7_9 cups-libs-1.6.3-52.el7_9 curl-7.29.0-59.el7_9.2 cyrus-sasl-lib-2.1.26-24.el7_9 dbus-1.10.24-15.el7 dbus-libs-1.10.24-15.el7 device-mapper-1.02.170-6.el7_9.5 device-mapper-event-1.02.170-6.el7_9.5 device-mapper-event-libs-1.02.170-6.el7_9.5 device-mapper-libs-1.02.170-6.el7_9.5 device-mapper-persistent-data-0.8.5-3.el7_9.2 dhclient-4.2.5-83.el7.centos.2 dhcp-common-4.2.5-83.el7.centos.2 dhcp-libs-4.2.5-83.el7.centos.2 diffutils-3.3-6.el7_9 dmidecode-3.2-5.el7_9.1 dracut-033-572.el7 dracut-network-033-572.el7 e2fsprogs-1.42.9-19.el7 e2fsprogs-libs-1.42.9-19.el7 elfutils-default-yama-scope-0.176-5.el7 elfutils-libelf-0.176-5.el7 elfutils-libs-0.176-5.el7 expat-2.1.0-15.el7_9 file-5.11-37.el7 file-libs-5.11-37.el7 firewalld-filesystem-0.6.3-13.el7_9 freetype-2.8-14.el7_9.1 gcc-4.8.5-44.el7 gcc-c++-4.8.5-44.el7 gd-2.0.35-27.el7_9 geoipupdate-2.5.0-2.el7 git-1.8.3.1-25.el7_9 glib2-2.56.1-9.el7_9 glibc-2.17-326.el7_9.3 glibc-common-2.17-326.el7_9.3 glibc-devel-2.17-326.el7_9.3 glibc-headers-2.17-326.el7_9.3 grub2-2.02-0.87.0.2.el7.centos.14 grub2-common-2.02-0.87.0.2.el7.centos.14 grub2-pc-2.02-0.87.0.2.el7.centos.14 grub2-pc-modules-2.02-0.87.0.2.el7.centos.14 grub2-tools-2.02-0.87.0.2.el7.centos.14 grub2-tools-extra-2.02-0.87.0.2.el7.centos.14 grub2-tools-minimal-2.02-0.87.0.2.el7.centos.14 gtk-update-icon-cache-3.22.30-8.el7_9 gzip-1.5-11.el7_9 haveged-1.9.13-1.el7 httpd-2.4.6-99.el7.centos.1 httpd-tools-2.4.6-99.el7.centos.1 hunspell-1.3.2-16.el7 hwdata-0.252-9.7.el7 initscripts-9.49.53-1.el7_9.1 iproute-4.11.0-30.el7 iptables-1.4.21-35.el7 iptables-services-1.4.21-35.el7 java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9 java-1.8.0-openjdk-headless-1.8.0.412.b08-1.el7_9 kbd-1.15.5-16.el7_9 kbd-legacy-1.15.5-16.el7_9 kbd-misc-1.15.5-16.el7_9 kernel-3.10.0-1160.119.1.el7 kernel-devel-3.10.0-1160.119.1.el7 kernel-headers-3.10.0-1160.119.1.el7 kernel-tools-3.10.0-1160.119.1.el7 kernel-tools-libs-3.10.0-1160.119.1.el7 kexec-tools-2.0.15-51.el7_9.3 kpartx-0.4.9-136.el7_9 krb5-libs-1.15.1-55.el7_9 less-458-10.el7_9 libX11-1.6.7-5.el7_9 libX11-common-1.6.7-5.el7_9 libXpm-3.5.12-2.el7_9 libblkid-2.23.2-65.el7_9.1 libcom_err-1.42.9-19.el7 libcroco-0.6.12-6.el7_9 libcurl-7.29.0-59.el7_9.2 libgcc-4.8.5-44.el7 libgomp-4.8.5-44.el7 libmount-2.23.2-65.el7_9.1 libpcap-1.5.3-13.el7_9 libpng-1.5.13-8.el7 librsync-2.3.4-1.el7 libsmartcols-2.23.2-65.el7_9.1 libss-1.42.9-19.el7 libssh2-1.8.0-4.el7_9.1 libstdc++-4.8.5-44.el7 libstdc++-devel-4.8.5-44.el7 libtiff-4.0.3-35.el7 libuuid-2.23.2-65.el7_9.1 libxml2-2.9.1-6.el7_9.6 libxml2-python-2.9.1-6.el7_9.6 libxslt-1.1.28-6.el7 linux-firmware-20200421-83.git78c0348.el7_9 lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7_9.1 lvm2-2.02.187-6.el7_9.5 lvm2-libs-2.02.187-6.el7_9.5 lz4-1.8.3-1.el7 mariadb-5.5.68-1.el7 mariadb-libs-5.5.68-1.el7 mariadb-server-5.5.68-1.el7 mdadm-4.1-9.el7_9 mesa-libEGL-18.3.4-12.el7_9 mesa-libGL-18.3.4-12.el7_9 mesa-libgbm-18.3.4-12.el7_9 mesa-libglapi-18.3.4-12.el7_9 mod_security_crs-2.2.9-3.el7_9 mod_ssl-2.4.6-99.el7.centos.1 munin-2.0.75-1.el7 munin-cgi-2.0.75-1.el7 munin-common-2.0.75-1.el7 munin-nginx-2.0.75-1.el7 munin-node-2.0.75-1.el7 ncdu-1.20-1.el7 net-snmp-5.7.2-49.el7_9.4 net-snmp-agent-libs-5.7.2-49.el7_9.4 net-snmp-libs-5.7.2-49.el7_9.4 net-snmp-utils-5.7.2-49.el7_9.4 nettle-2.7.1-9.el7_9 nspr-4.35.0-1.el7_9 nss-3.90.0-2.el7_9 nss-pem-1.0.3-7.el7_9.1 nss-softokn-3.90.0-6.el7_9 nss-softokn-freebl-3.90.0-6.el7_9 nss-sysinit-3.90.0-2.el7_9 nss-tools-3.90.0-2.el7_9 nss-util-3.90.0-1.el7_9 ntp-4.2.6p5-29.el7.centos.2 ntpdate-4.2.6p5-29.el7.centos.2 openldap-2.4.44-25.el7_9 openssh-7.4p1-23.el7_9 openssh-clients-7.4p1-23.el7_9 openssh-server-7.4p1-23.el7_9 openssl-1.0.2k-26.el7_9 openssl-libs-1.0.2k-26.el7_9 perl-5.16.3-299.el7_9 perl-DBD-Pg-2.19.3-5.el7_9 perl-Git-1.8.3.1-25.el7_9 perl-Net-Server-2.007-3.el7 perl-Pod-Escapes-1.04-299.el7_9 perl-libs-5.16.3-299.el7_9 perl-macros-5.16.3-299.el7_9 plymouth-0.8.9-0.34.20140113.el7.centos plymouth-core-libs-0.8.9-0.34.20140113.el7.centos plymouth-scripts-0.8.9-0.34.20140113.el7.centos polkit-0.112-26.el7_9.1 postgresql-libs-9.2.24-9.el7_9 procps-ng-3.3.10-28.el7 psmisc-22.20-17.el7 python-2.7.5-94.el7_9 python-devel-2.7.5-94.el7_9 python-libs-2.7.5-94.el7_9 python-perf-3.10.0-1160.119.1.el7 python-requests-2.6.0-10.el7 python-rpm-macros-3-34.el7 python-srpm-macros-3-34.el7 python-virtualenv-15.1.0-7.el7_9 python2-distro-1.5.0-1.el7 python2-pyrfc3339-1.1-3.el7 python2-rpm-macros-3-34.el7 python2-uritemplate-3.0.1-1.el7 rpm-4.11.3-48.el7_9 rpm-build-libs-4.11.3-48.el7_9 rpm-libs-4.11.3-48.el7_9 rpm-python-4.11.3-48.el7_9 rsync-3.1.2-12.el7_9 rsyslog-8.24.0-57.el7_9.3 screen-4.1.0-0.27.20120314git3c2946.el7_9 sed-4.2.2-7.el7 selinux-policy-3.13.1-268.el7_9.2 selinux-policy-targeted-3.13.1-268.el7_9.2 ssldump-1.8-1.el7 subversion-1.7.14-16.el7 subversion-libs-1.7.14-16.el7 sudo-1.8.23-10.el7_9.3 sysstat-10.1.5-20.el7_9 systemd-219-78.el7_9.9 systemd-libs-219-78.el7_9.9 systemd-sysv-219-78.el7_9.9 telnet-0.17-66.el7 tuned-2.11.0-12.el7_9 tzdata-2024a-1.el7 tzdata-java-2024a-1.el7 unzip-6.0-24.el7_9 util-linux-2.23.2-65.el7_9.1 vim-common-7.4.629-8.el7_9 vim-enhanced-7.4.629-8.el7_9 vim-filesystem-7.4.629-8.el7_9 vim-minimal-7.4.629-8.el7_9 virt-what-1.18-4.el7_9.1 webmin-2.201-1 xfsprogs-4.5.0-22.el7 xz-5.2.2-2.el7_9 xz-libs-5.2.2-2.el7_9 yum-3.4.3-168.el7.centos yum-cron-3.4.3-168.el7.centos yum-plugin-fastestmirror-1.1.31-54.el7_8 yum-utils-1.1.31-54.el7_8 zlib-1.2.7-21.el7_9 &> "${tmpDir}/update.log"

# log the post-state packages and versions
time nice rpm -qa &> "${tmpDir}/after.log"

# check to see what changes require a reboot, if any
time nice needs-restarting &> "${tmpDir}/needs-restarting.log"
time nice needs-restarting -r &> "${tmpDir}/needs-reboot.log"

# reboot the system to apply kernel changes
reboot

# after reboot, initiate a new backup
time sudo /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

Post-state Proof

The below command will output zero or nearly-zero packages after this CHG is successfully complete 

yum list updates

Validation Steps

  1. Access the following URLs; make sure redirects and the destination pages look sane
    1. http://wiki.opensourceecology.org/
    2. http://opensourceecology.org/
    3. http://openbuildinginstitute.org/
    4. http://store.opensourceecology.org/
    5. http://oswh.opensourceecology.org/
    6. http://fef.opensourceecology.org/
    7. http://microfactory.opensourceecology.org/
    8. http://forum.opensourceecology.org/
    9. http://phplist.opensourceecology.org/lists/
  1. Access and login to the following private sites too to make sure everything is sane
    1. https://awstats.opensourceecology.org:4443
    2. https://munin.opensourceecology.org:4443
  1. Login and attempt to make a trivial change on the following sites
    1. https://wiki.opensourceecology.org/
    2. https://opensourceecology.org/
  1. Test a backups run; make sure they're successful and uploaded to backblaze

Revert Steps

All the packages can be reverted to their previous versions using the following command 

yum update-to TODO

See Also

  1. https://serverfault.com/questions/1014455/how-to-list-packages-needing-update-in-format-for-yum-update-to
  2. CHG-2020-05-04_yum_update Last yum update (in May 2020)
  3. List of other CHG "tickets"
Retrieved from "https://wiki.opensourceecology.org/index.php?title=CHG-2024-07-26_yum_update&oldid=298905"