Maltfield Log/2020 Q2

From Open Source Ecology
Jump to navigation Jump to search

My work log from the year 2020 Quarter 2. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

  1. Maltfield_Log
  2. User:Maltfield
  3. Special:Contributions/Maltfield

Thr Apr 23, 2020

  1. since yesterday, it appears that the sudo package has been automatically updated by `unattended-upgrades` triggered by our cron job. Horray!
root@osestaging1-discourse-ose:/etc/nginx# dpkg -l | grep -i sudo
ii  sudo                            1.8.27-1+deb10u2             amd64        Provide limited super user privileges to specific users
root@osestaging1-discourse-ose:/etc/nginx#
  1. My cron job was set to trigger it to go off a 04:20
root@osestaging1-discourse-ose:/var/log/nginx# cat /etc/cron.d/unattended-upgrades 
################################################################################
# File:    /etc/cron.d/unattended-upgrades
# Version: 0.1
# Purpose: run unattended-upgrades in lieu of systemd. For more info see
#           * https://wiki.opensourceecology.org/wiki/Discourse
#           * https://meta.discourse.org/t/does-discourse-container-use-unattended-upgrades/136296/3
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2020-03-23
# Updated: 2020-03-23
################################################################################
20 04 * * * root /usr/bin/nice /usr/bin/unattended-upgrades --debug




root@osestaging1-discourse-ose:/var/log/nginx#
  1. sure enough, there's an unattended-upgrades log with an entry starting at 04:20. Looks like it installed a hell of a lot more security-critical updates than just sudo, though
==> /var/log/unattended-upgrades/unattended-upgrades.log <==
2020-04-23 04:20:03,462 INFO Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
2020-04-23 04:20:03,524 INFO Initial blacklist :
2020-04-23 04:20:03,524 INFO Initial whitelist:
2020-04-23 04:20:03,524 INFO Starting unattended upgrades script
2020-04-23 04:20:03,524 INFO Allowed origins are: origin=Debian,codename=buster,label=Debian, origin=Debian,codename=buster,label=Debian-Security
2020-04-23 04:20:06,461 DEBUG Using (^linux-image-[0-9]+\.[0-9\.]+-.*|^linux-headers-[0-9]+\.[0-9\.]+-.*|^linux-image-extra-[0-9]+\.[0-9\.]+-.*|^linux-modules-[0-9]+\.[0-9\.]+-.*|^linux-modules-extra-[0-9]+\.[0-9\.]+-.*|^linux-signed-image-[0-9]+\.[0-9\.]+-.*|^linux-image-unsigned-[0-9]+\.[0-9\.]+-.*|^kfreebsd-image-[0-9]+\.[0-9\.]+-.*|^kfreebsd-headers-[0-9]+\.[0-9\.]+-.*|^gnumach-image-[0-9]+\.[0-9\.]+-.*|^.*-modules-[0-9]+\.[0-9\.]+-.*|^.*-kernel-[0-9]+\.[0-9\.]+-.*|^linux-backports-modules-.*-[0-9]+\.[0-9\.]+-.*|^linux-modules-.*-[0-9]+\.[0-9\.]+-.*|^linux-tools-[0-9]+\.[0-9\.]+-.*|^linux-cloud-tools-[0-9]+\.[0-9\.]+-.*|^linux-buildinfo-[0-9]+\.[0-9\.]+-.*|^linux-source-[0-9]+\.[0-9\.]+-.*) regexp to find kernel packages
2020-04-23 04:20:06,481 DEBUG Using (^linux-image-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-headers-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-image-extra-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-modules-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-modules-extra-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-signed-image-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-image-unsigned-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^kfreebsd-image-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^kfreebsd-headers-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^gnumach-image-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^.*-modules-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^.*-kernel-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-backports-modules-.*-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-modules-.*-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-tools-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-cloud-tools-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-buildinfo-3\.10\.0\-957\.21\.3\.el7\.x86_64$|^linux-source-3\.10\.0\-957\.21\.3\.el7\.x86_64$) regexp to find running kernel packages
2020-04-23 04:20:07,539 DEBUG Checking: git ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:07,793 DEBUG Checking: git-man ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:08,054 DEBUG Checking: icu-devtools ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:08,415 DEBUG Checking: libgnutls-dane0 ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:08,597 DEBUG Checking: libgnutls30 ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:08,811 DEBUG Checking: libicu-dev ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:09,000 DEBUG Checking: libicu63 ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:09,314 DEBUG Checking: libssl-dev ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:09,566 DEBUG Checking: libssl1.1 ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:09,819 DEBUG Checking: nodejs ([<Origin component:'main' archive:'' origin:'Node Source' label:'Node Source' site:'deb.nodesource.com' isTrusted:True>])
2020-04-23 04:20:09,821 DEBUG adjusting candidate version: nodejs=10.15.2~dfsg-2
2020-04-23 04:20:09,855 DEBUG Checking: openssl ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian-Security' site:'security.debian.org' isTrusted:True>])
2020-04-23 04:20:10,330 DEBUG Checking: sudo ([<Origin component:'main' archive:'stable' origin:'Debian' label:'Debian' site:'deb.debian.org' isTrusted:True>])
2020-04-23 04:20:11,364 DEBUG pkgs that look like they should be upgraded: git
git-man
icu-devtools
libgnutls-dane0
libgnutls30
libicu-dev
libicu63
libssl-dev
libssl1.1
openssl
sudo
2020-04-23 04:20:14,287 DEBUG fetch.run() result: 0
2020-04-23 04:20:14,863 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 316560 DestFile:'/var/cache/apt/archives/libgnutls-dane0_3.6.7-4+deb10u3_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/g/gnutls28/libgnutls-dane0_3.6.7-4+deb10u3_amd64.deb' ID:2 ErrorText: ''>
2020-04-23 04:20:14,875 DEBUG check_conffile_prompt(/var/cache/apt/archives/libgnutls-dane0_3.6.7-4+deb10u3_amd64.deb)
2020-04-23 04:20:14,877 DEBUG found pkg: libgnutls-dane0
2020-04-23 04:20:14,878 DEBUG No conffiles in deb /var/cache/apt/archives/libgnutls-dane0_3.6.7-4+deb10u3_amd64.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,879 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 1124428 DestFile:'/var/cache/apt/archives/libgnutls30_3.6.7-4+deb10u3_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/g/gnutls28/libgnutls30_3.6.7-4+deb10u3_amd64.deb' ID:3 ErrorText: ''>
2020-04-23 04:20:14,879 DEBUG check_conffile_prompt(/var/cache/apt/archives/libgnutls30_3.6.7-4+deb10u3_amd64.deb)
2020-04-23 04:20:14,880 DEBUG found pkg: libgnutls30
2020-04-23 04:20:14,884 DEBUG No conffiles in deb /var/cache/apt/archives/libgnutls30_3.6.7-4+deb10u3_amd64.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,891 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 1793184 DestFile:'/var/cache/apt/archives/libssl-dev_1.1.1d-0+deb10u3_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/o/openssl/libssl-dev_1.1.1d-0+deb10u3_amd64.deb' ID:4 ErrorText: ''>
2020-04-23 04:20:14,891 DEBUG check_conffile_prompt(/var/cache/apt/archives/libssl-dev_1.1.1d-0+deb10u3_amd64.deb)
2020-04-23 04:20:14,893 DEBUG found pkg: libssl-dev
2020-04-23 04:20:14,894 DEBUG No conffiles in deb /var/cache/apt/archives/libssl-dev_1.1.1d-0+deb10u3_amd64.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,894 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 1538460 DestFile:'/var/cache/apt/archives/libssl1.1_1.1.1d-0+deb10u3_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/o/openssl/libssl1.1_1.1.1d-0+deb10u3_amd64.deb' ID:5 ErrorText: ''>
2020-04-23 04:20:14,894 DEBUG check_conffile_prompt(/var/cache/apt/archives/libssl1.1_1.1.1d-0+deb10u3_amd64.deb)
2020-04-23 04:20:14,900 DEBUG found pkg: libssl1.1
2020-04-23 04:20:14,911 DEBUG No conffiles in deb /var/cache/apt/archives/libssl1.1_1.1.1d-0+deb10u3_amd64.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,912 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 1620352 DestFile:'/var/cache/apt/archives/git-man_1%3a2.20.1-2+deb10u3_all.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/g/git/git-man_2.20.1-2+deb10u3_all.deb' ID:6 ErrorText: ''>
2020-04-23 04:20:14,912 DEBUG check_conffile_prompt(/var/cache/apt/archives/git-man_1%3a2.20.1-2+deb10u3_all.deb)
2020-04-23 04:20:14,920 DEBUG found pkg: git-man
2020-04-23 04:20:14,923 DEBUG No conffiles in deb /var/cache/apt/archives/git-man_1%3a2.20.1-2+deb10u3_all.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,923 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 5633060 DestFile:'/var/cache/apt/archives/git_1%3a2.20.1-2+deb10u3_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/g/git/git_2.20.1-2+deb10u3_amd64.deb' ID:7 ErrorText: ''>
2020-04-23 04:20:14,923 DEBUG check_conffile_prompt(/var/cache/apt/archives/git_1%3a2.20.1-2+deb10u3_amd64.deb)
2020-04-23 04:20:14,925 DEBUG found pkg: git
2020-04-23 04:20:14,935 DEBUG conffile line: /etc/bash_completion.d/git-prompt 7baac5c3ced94ebf2c0e1dde65c3b1a6
2020-04-23 04:20:14,938 DEBUG current md5: 7baac5c3ced94ebf2c0e1dde65c3b1a6
2020-04-23 04:20:14,938 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 9186164 DestFile:'/var/cache/apt/archives/libicu-dev_63.1-6+deb10u1_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/i/icu/libicu-dev_63.1-6+deb10u1_amd64.deb' ID:8 ErrorText: ''>
2020-04-23 04:20:14,938 DEBUG check_conffile_prompt(/var/cache/apt/archives/libicu-dev_63.1-6+deb10u1_amd64.deb)
2020-04-23 04:20:14,940 DEBUG found pkg: libicu-dev
2020-04-23 04:20:14,942 DEBUG No conffiles in deb /var/cache/apt/archives/libicu-dev_63.1-6+deb10u1_amd64.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,943 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 188624 DestFile:'/var/cache/apt/archives/icu-devtools_63.1-6+deb10u1_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/i/icu/icu-devtools_63.1-6+deb10u1_amd64.deb' ID:9 ErrorText: ''>
2020-04-23 04:20:14,950 DEBUG check_conffile_prompt(/var/cache/apt/archives/icu-devtools_63.1-6+deb10u1_amd64.deb)
2020-04-23 04:20:14,951 DEBUG found pkg: icu-devtools
2020-04-23 04:20:14,953 DEBUG No conffiles in deb /var/cache/apt/archives/icu-devtools_63.1-6+deb10u1_amd64.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,953 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 8300324 DestFile:'/var/cache/apt/archives/libicu63_63.1-6+deb10u1_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/i/icu/libicu63_63.1-6+deb10u1_amd64.deb' ID:10 ErrorText: ''>
2020-04-23 04:20:14,953 DEBUG check_conffile_prompt(/var/cache/apt/archives/libicu63_63.1-6+deb10u1_amd64.deb)
2020-04-23 04:20:14,954 DEBUG found pkg: libicu63
2020-04-23 04:20:14,956 DEBUG No conffiles in deb /var/cache/apt/archives/libicu63_63.1-6+deb10u1_amd64.deb (There is no member named 'conffiles')
2020-04-23 04:20:14,956 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 844404 DestFile:'/var/cache/apt/archives/openssl_1.1.1d-0+deb10u3_amd64.deb' DescURI: 'http://security.debian.org/debian-security/pool/updates/main/o/openssl/openssl_1.1.1d-0+deb10u3_amd64.deb' ID:11 ErrorText: ''>
2020-04-23 04:20:14,957 DEBUG check_conffile_prompt(/var/cache/apt/archives/openssl_1.1.1d-0+deb10u3_amd64.deb)
2020-04-23 04:20:14,960 DEBUG found pkg: openssl
2020-04-23 04:20:14,961 DEBUG conffile line: /etc/ssl/openssl.cnf 8dd4de5642570c91e9071900b6b9d5bc
2020-04-23 04:20:14,971 DEBUG current md5: 8dd4de5642570c91e9071900b6b9d5bc
2020-04-23 04:20:14,971 DEBUG <apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 1244824 DestFile:'/var/cache/apt/archives/sudo_1.8.27-1+deb10u2_amd64.deb' DescURI: 'http://deb.debian.org/debian/pool/main/s/sudo/sudo_1.8.27-1+deb10u2_amd64.deb' ID:1 ErrorText: ''>
2020-04-23 04:20:14,971 DEBUG check_conffile_prompt(/var/cache/apt/archives/sudo_1.8.27-1+deb10u2_amd64.deb)
2020-04-23 04:20:14,974 DEBUG found pkg: sudo
2020-04-23 04:20:14,975 DEBUG conffile line: /etc/init.d/sudo 1153f6e6fa7c0e2166779df6ad43f1a8
2020-04-23 04:20:14,993 DEBUG current md5: 1153f6e6fa7c0e2166779df6ad43f1a8
2020-04-23 04:20:14,993 DEBUG conffile line: /etc/pam.d/sudo 85da64f888739f193fc0fa896680030e
2020-04-23 04:20:15,002 DEBUG current md5: 85da64f888739f193fc0fa896680030e
2020-04-23 04:20:15,002 DEBUG conffile line: /etc/sudoers 45437b4e86fba2ab890ac81db2ec3606
2020-04-23 04:20:15,005 DEBUG current md5: 45437b4e86fba2ab890ac81db2ec3606
2020-04-23 04:20:15,005 DEBUG conffile line: /etc/sudoers.d/README 8d3cf36d1713f40a0ddc38e1b21a51b6
2020-04-23 04:20:15,011 DEBUG current md5: 8d3cf36d1713f40a0ddc38e1b21a51b6
2020-04-23 04:20:15,011 DEBUG blacklist: []
2020-04-23 04:20:15,011 DEBUG whitelist: []
2020-04-23 04:20:15,011 INFO Packages that will be upgraded: git git-man icu-devtools libgnutls-dane0 libgnutls30 libicu-dev libicu63 libssl-dev libssl1.1 openssl sudo
2020-04-23 04:20:15,027 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2020-04-23 04:20:15,419 DEBUG applying set ['libicu63', 'libicu-dev', 'icu-devtools']
...
2020-04-23 04:20:31,192 DEBUG left to upgrade {'libssl-dev', 'sudo', 'git-man', 'libssl1.1', 'openssl', 'git
', 'libgnutls-dane0', 'libgnutls30'}
2020-04-23 04:20:31,473 DEBUG applying set ['libssl-dev', 'libssl1.1']
...
2020-04-23 04:20:40,854 DEBUG left to upgrade {'sudo', 'git-man', 'openssl', 'git', 'libgnutls-dane0', 'libg
nutls30'}
2020-04-23 04:20:41,256 DEBUG applying set ['sudo']
...
2020-04-23 04:20:45,311 DEBUG left to upgrade {'git-man', 'openssl', 'git', 'libgnutls-dane0', 'libgnutls30'
}
2020-04-23 04:20:45,535 DEBUG applying set ['git-man']
...
2020-04-23 04:20:48,234 DEBUG left to upgrade {'libgnutls-dane0', 'openssl', 'libgnutls30', 'git'}
2020-04-23 04:20:48,368 DEBUG applying set ['openssl']
...
2020-04-23 04:20:51,259 DEBUG left to upgrade {'libgnutls-dane0', 'libgnutls30', 'git'}
2020-04-23 04:20:51,453 DEBUG applying set ['git']
...
2020-04-23 04:20:56,727 DEBUG left to upgrade {'libgnutls-dane0', 'libgnutls30'}
2020-04-23 04:20:56,897 DEBUG applying set ['libgnutls30', 'libgnutls-dane0']
...
2020-04-23 04:20:59,859 DEBUG left to upgrade set()
2020-04-23 04:20:59,860 INFO All upgrades installed
2020-04-23 04:21:00,337 DEBUG InstCount=0 DelCount=0 BrokenCount=0
2020-04-23 04:21:00,356 DEBUG Extracting content from /var/log/unattended-upgrades/unattended-upgrades-dpkg.
log since 2020-04-23 04:20:03

==> /var/log/unattended-upgrades/unattended-upgrades-dpkg.log <==
Log started: 2020-04-23  04:20:15
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48062 files and directories currently installed.)
Preparing to unpack .../libicu-dev_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu-dev:amd64 (63.1-6+deb10u1) over (63.1-6) ...
Preparing to unpack .../icu-devtools_63.1-6+deb10u1_amd64.deb ...
Unpacking icu-devtools (63.1-6+deb10u1) over (63.1-6) ...
Preparing to unpack .../libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) over (63.1-6) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up icu-devtools (63.1-6+deb10u1) ...
Setting up libicu-dev:amd64 (63.1-6+deb10u1) ...
Processing triggers for libc-bin (2.28-10) ...
Log ended: 2020-04-23  04:20:29
...
Log started: 2020-04-23  04:20:31
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48062 files and directories currently installed.)
Preparing to unpack .../libssl-dev_1.1.1d-0+deb10u3_amd64.deb ...
Unpacking libssl-dev:amd64 (1.1.1d-0+deb10u3) over (1.1.1d-0+deb10u2) ...
Preparing to unpack .../libssl1.1_1.1.1d-0+deb10u3_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1d-0+deb10u3) over (1.1.1d-0+deb10u2) ...
Setting up libssl1.1:amd64 (1.1.1d-0+deb10u3) ...
Setting up libssl-dev:amd64 (1.1.1d-0+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
Log ended: 2020-04-23  04:20:38
...
Log started: 2020-04-23  04:20:41
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48062 files and directories currently installed.)
Preparing to unpack .../sudo_1.8.27-1+deb10u2_amd64.deb ...
Unpacking sudo (1.8.27-1+deb10u2) over (1.8.27-1+deb10u1) ...
Setting up sudo (1.8.27-1+deb10u2) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.
Processing triggers for systemd (241-7~deb10u3) ...
Log ended: 2020-04-23  04:20:43
...
Log started: 2020-04-23  04:20:45
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48062 files and directories currently installed.)
Preparing to unpack .../git-man_1%3a2.20.1-2+deb10u3_all.deb ...
Unpacking git-man (1:2.20.1-2+deb10u3) over (1:2.20.1-2+deb10u1) ...
Setting up git-man (1:2.20.1-2+deb10u3) ...
Log ended: 2020-04-23  04:20:46
...
Log started: 2020-04-23  04:20:48
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48062 files and directories currently installed.)
Preparing to unpack .../openssl_1.1.1d-0+deb10u3_amd64.deb ...
Unpacking openssl (1.1.1d-0+deb10u3) over (1.1.1d-0+deb10u2) ...
Setting up openssl (1.1.1d-0+deb10u3) ...
Log ended: 2020-04-23  04:20:49
...
Log started: 2020-04-23  04:20:51
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48062 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.20.1-2+deb10u3_amd64.deb ...
Unpacking git (1:2.20.1-2+deb10u3) over (1:2.20.1-2+deb10u1) ...
Setting up git (1:2.20.1-2+deb10u3) ...
Log ended: 2020-04-23  04:20:55
...Log started: 2020-04-23  04:20:57
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 48070 files and directories currently installed.)
Preparing to unpack .../libgnutls-dane0_3.6.7-4+deb10u3_amd64.deb ...
Unpacking libgnutls-dane0:amd64 (3.6.7-4+deb10u3) over (3.6.7-4+deb10u2) ...
Preparing to unpack .../libgnutls30_3.6.7-4+deb10u3_amd64.deb ...
Unpacking libgnutls30:amd64 (3.6.7-4+deb10u3) over (3.6.7-4+deb10u2) ...
Setting up libgnutls30:amd64 (3.6.7-4+deb10u3) ...
Setting up libgnutls-dane0:amd64 (3.6.7-4+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
Log ended: 2020-04-23  04:20:58
  1. Interestingly, it appears to have gotten kicked-off at 08:00 again somehow *shurg*
==> /var/log/unattended-upgrades/unattended-upgrades.log <==
2020-04-23 08:00:15,117 INFO Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
2020-04-23 08:00:15,125 INFO Initial blacklist :
2020-04-23 08:00:15,125 INFO Initial whitelist:
2020-04-23 08:00:15,126 INFO Starting unattended upgrades script
2020-04-23 08:00:15,126 INFO Allowed origins are: origin=Debian,codename=buster,label=Debian, origin=Debian,codename=buster,label=Debian-Security
2020-04-23 08:00:20,219 INFO Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
2020-04-23 08:00:20,222 INFO Initial blacklist :
2020-04-23 08:00:20,223 INFO Initial whitelist:
2020-04-23 08:00:20,223 INFO Starting unattended upgrades script
2020-04-23 08:00:20,223 INFO Allowed origins are: origin=Debian,codename=buster,label=Debian, origin=Debian,codename=buster,label=Debian-Security
2020-04-23 08:00:24,157 INFO No packages found that can be upgraded unattended and no pending auto-removals
  1. I documented my solution to the cron bug here https://meta.discourse.org/t/cron-in-docker-container-cannot-make-remove-an-entry-for-the-specified-session/148969/2
  2. And I also documented my solution to fixing the `unattended-upgrades` install with cron here https://meta.discourse.org/t/does-discourse-container-use-unattended-upgrades/136296/9?u=maltfield
  3. Finally, I crossed-off the 'unattended-upgrades' requirement from the TODO list!
  1. ...
  1. the last item on the TODO list that I haven't even begun yet is the hardened file permissions.
  2. First I just searched their forums (since there's no fucking documentation for Discourse)
    1. I found this CentOS install guide; it only said to set the owner to 'discourse' for the whole '/var/discourse/' dir. For us it's currently root:root; we don't even have a 'discourse' user on the box https://meta.discourse.org/t/how-to-install-discourse-on-an-isolated-centos-7-server/73538/22
  3. I couldn't find anything else, so I posted a topic on it https://meta.discourse.org/t/minimum-hardened-file-permissions/148974
  1. ...
  1. Meanwhile, I'm going to test the upgrade documentation again https://wiki.opensourceecology.org/wiki/Discourse#Updating_Discoruse
  2. I had an update conflict with `install-nginx` again, but the documentation was clear enough for me to resolve this and continue
  3. I made some slight changes to the upgrade documentation, but otherwise it was smooth.
  4. I'm crossing-off the "Test/document Discourse upgrade process" TODO item!
  1. ..
  1. Now, I'm going to do a fresh install following my guide.
  2. I already have a backup from the upgrade I just did
[root@osestaging1 base]# echo $tmpDir
/var/tmp/discourseUpgrade.20200423_11:15:32
[root@osestaging1 base]# ls -lah /var/tmp/discourseUpgrade.20200423_11\:15\:32/
total 184M
drwx------.  2 root      root      4.0K Apr 23 11:16 .
drwxrwxrwt. 59 root      root       12K Apr 23 12:13 ..
-rw-r--r--.  1 tgriffing tgriffing  70M Apr 23 11:16 discourse-2020-04-23-111614-v20200417183143.tar.gz
-rw-r--r--.  1 root      root      114M Apr 23 11:16 discourse_files.20200423_11:15:32.tar.gz
[root@osestaging1 base]#
  1. So now I'm going to clobber the staging server's Discourse install by doing a destructive sync from prod to staging
user@ose:~$ ssh oseprod
Last login: Mon Mar 16 06:58:53 2020 from 110-44-121-46.vianet.com.np
[maltfield@opensourceecology ~]$ screen -dr syncToSt
...
[maltfield@opensourceecology ~]$ sudo su -
[sudo] password for maltfield: 
Last login: Mon Mar 16 06:59:11 UTC 2020 on pts/34
[root@opensourceecology ~]# time nice /root/bin/syncToStaging.sh &> /var/log/syncToStaging.log
  1. I fixed some minor issues with the documentation's smtp section
  2. I updated the "inner nginx" config section to include the changes for varnish
  3. oh fuck, looks like I didn't actually save my varnish config before I wiped it. Fortunately I did log it https://wiki.opensourceecology.org/wiki/Maltfield_Log/2020_Q1#Tue_Mar_23.2C_2020
  4. ugh, apparently I did the same thing to my "outer" ngingx config file. I'll update the wiki install guide now with these, but this will have to be fully tested again..
  5. ok, I made some other documentation changes to the install guide too
  6. I was able to restore and access the site successfully!
  7. Now the only item left is the permissions/ownership of the files/dirs. I'll tackle that and do another whole-install process next week.

Wed Apr 22, 2020

  1. I'm returning to some OSE work after a brief segway building-out an open-source COVID-19-related project
    1. https://www.coviz.org/
  2. ...
  3. unfortunately it appears that our anacron *still* hasn't kicked-off the upgrade of sudo via unattended-upgrades
root@osestaging1-discourse-ose:/var/www/discourse# dpkg -l | grep -i sudo
ii  sudo                            1.8.27-1+deb10u1             amd64        Provide limited super user privileges to specific users
root@osestaging1-discourse-ose:/var/www/discourse#
  1. syslog was flooded with these messages all the way up to the top of the `screen` scrollback
root@osestaging1-discourse-ose:/var/www/discourse# tail -f /var/log/syslog /var/log/unattended-upgrades/*
==> /var/log/syslog <==
Apr 22 09:35:01 osestaging1-discourse-ose CRON[21770]: Cannot make/remove an entry for the specified session
Apr 22 09:45:01 osestaging1-discourse-ose CRON[22402]: Cannot make/remove an entry for the specified session
Apr 22 09:55:01 osestaging1-discourse-ose CRON[23038]: Cannot make/remove an entry for the specified session
Apr 22 10:05:01 osestaging1-discourse-ose CRON[23673]: Cannot make/remove an entry for the specified session
Apr 22 10:15:01 osestaging1-discourse-ose CRON[24307]: Cannot make/remove an entry for the specified session
Apr 22 10:17:01 osestaging1-discourse-ose CRON[24440]: Cannot make/remove an entry for the specified session
Apr 22 10:25:01 osestaging1-discourse-ose CRON[24947]: Cannot make/remove an entry for the specified session
Apr 22 10:35:01 osestaging1-discourse-ose CRON[25584]: Cannot make/remove an entry for the specified session
Apr 22 10:45:01 osestaging1-discourse-ose CRON[26215]: Cannot make/remove an entry for the specified session
Apr 22 10:55:01 osestaging1-discourse-ose CRON[26857]: Cannot make/remove an entry for the specified session
  1. I confirmed that my anacron is in-place
root@osestaging1-discourse-ose:/var/www/discourse# cat /etc/cron.daily/unattended-upgrades 
#!/bin/bash
################################################################################
# File:    /etc/cron.daily/unattended-upgrades
# Version: 0.1
# Purpose: run unattended-upgrades in lieu of systemd. For more info see
#           * https://wiki.opensourceecology.org/wiki/Discourse
#           * https://meta.discourse.org/t/does-discourse-container-use-unattended-upgrades/136296/3
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2020-03-23
# Updated: 2020-03-23
################################################################################
/usr/bin/nice /usr/bin/unattended-upgrades --debug




root@osestaging1-discourse-ose:/var/www/discourse#
  1. But I don't see an anacron service running..
root@osestaging1-discourse-ose:/var/www/discourse# ps -ef | grep -i cron
root       721   715  0 Mar30 ?        00:00:00 runsv cron
root       727   721  0 Mar30 ?        00:00:16 cron -f
root     27421  1204  0 11:03 pts/1    00:00:00 grep -i cron
root@osestaging1-discourse-ose:/var/www/discourse#
  1. though, of course, anacron is supposed to be started by regular cron
root@osestaging1-discourse-ose:/var/www/discourse# cat /etc/cron.d/anacron 
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

30 7    * * *   root    /usr/sbin/anacron -s >/dev/null
root@osestaging1-discourse-ose:/var/www/discourse#
  1. I manually started anacron; we'll see if that does something
root@osestaging1-discourse-ose:/var/www/discourse# /usr/sbin/anacron -s >/dev/null
root@osestaging1-discourse-ose:/var/www/discourse# ps -ef | grep -i cron
root       721   715  0 Mar30 ?        00:00:00 runsv cron
root       727   721  0 Mar30 ?        00:00:16 cron -f
root     30691     0  0 11:54 ?        00:00:00 /usr/sbin/anacron -s
root     30698  1204  0 11:55 pts/1    00:00:00 grep -i cron
root@osestaging1-discourse-ose:/var/www/discourse#
  1. I could see from the logs this time that anacron's daily jobs got kicked-off, but still sudo didn't get updated
==> /var/log/syslog <==
Apr 22 11:45:01 osestaging1-discourse-ose CRON[30055]: Cannot make/remove an entry for the specified session
Apr 22 11:54:59 osestaging1-discourse-ose anacron[30691]: Anacron 2.3 started on 2020-04-22
Apr 22 11:54:59 osestaging1-discourse-ose anacron[30691]: Will run job `cron.daily' in 5 min.
Apr 22 11:54:59 osestaging1-discourse-ose anacron[30691]: Will run job `cron.weekly' in 10 min.
Apr 22 11:54:59 osestaging1-discourse-ose anacron[30691]: Will run job `cron.monthly' in 15 min.
Apr 22 11:54:59 osestaging1-discourse-ose anacron[30691]: Jobs will be executed sequentially
Apr 22 11:55:01 osestaging1-discourse-ose CRON[30694]: Cannot make/remove an entry for the specified session
Apr 22 11:59:59 osestaging1-discourse-ose anacron[30691]: Job `cron.daily' started
Apr 22 11:59:59 osestaging1-discourse-ose anacron[31015]: Updated timestamp for job `cron.daily' to 2020-04-22
Apr 22 12:05:01 osestaging1-discourse-ose CRON[31345]: Cannot make/remove an entry for the specified session
...
root@osestaging1-discourse-ose:/var/www/discourse# dpkg -l | grep -i sudo
ii  sudo                            1.8.27-1+deb10u1             amd64        Provide limited super user privileges to specific users
root@osestaging1-discourse-ose:/var/www/discourse#
  1. I commented out that 'session required pam_loginid.so' line that I've read issues about and re-started anacron


  1. holy shit that worked


root@osestaging1-discourse-ose:/var/www/discourse# dpkg -l | grep -i sudo
ii  sudo                            1.8.27-1+deb10u2             amd64        Provide limited super user privileges to specific users
root@osestaging1-discourse-ose:/var/www/discourse# 



  1. apparently this was a bug that got fixed in Debain recently, but apparently not our Discourse's debian build https://stackoverflow.com/questions/43323754/cannot-make-remove-an-entry-for-the-specified-session-cron
    2. 

  2. I rigged up a sed command for this fix and added it to the Docker install documentation https://wiki.opensourceecology.org/wiki/Discourse#unattended-upgrades
    3. 

  3. I did a rebuild of the Discourse docker image (including fixing this damn stuck docker container that tells me docker isn't good for production apps at all)


[root@osestaging1 discourse]# time /var/discourse/launcher rebuild discourse_ose
...
2020-04-22 12:54:22.823 UTC [60] LOG:  shutting down
173:M 22 Apr 2020 12:54:22.832 * DB saved on disk
173:M 22 Apr 2020 12:54:22.832 # Redis is now ready to exit, bye bye...
2020-04-22 12:54:22.922 UTC [56] LOG:  database system is shut down
sha256:24e9ff23984a280d4abdac05971d447f55868c477c542b4e9ae7c9b3c15715d4
6cd93242ddfbf37e847697ac3323623a71ea24c1595972e694dcd7b5e990da95
Removing old container
+ /bin/docker rm discourse_ose
Error response from daemon: container 15a32ba3c8e485f9591c7925dcd48ee44ca0216e4df99570a29e3b04990267dd: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3a9100ce8740fe74bf3e6d0f1ecf6ea9d5ce97a1270a1fc36348f1ddb60e51ab/merged: device or resource busy

starting up existing container
+ /bin/docker start discourse_ose
Error response from daemon: container is marked for removal and cannot be started
Error: failed to start containers: discourse_ose

real    8m32.751s
user    0m1.529s
sys     0m1.528s
[root@osestaging1 discourse]#



  1. And I downgraded the app


root@osestaging1-discourse-ose:/var/www/discourse# logout
[root@osestaging1 sites-enabled]# /var/discourse/launcher enter discourse_ose
root@osestaging1-discourse-ose:/var/www/discourse# dpkg -l | grep -i sudo
ii  sudo                            1.8.27-1+deb10u2             amd64        Provide limited super user privileges to specific users
root@osestaging1-discourse-ose:/var/www/discourse# 



  1. finally, to test this, I re-downgraded the sudo package to the insecure version


root@osestaging1-discourse-ose:/var/www/discourse# apt-get install sudo=1.8.27-1+deb10u1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be DOWNGRADED:
  sudo
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 11 not upgraded.
Need to get 1,244 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 sudo amd64 1.8.27-1+deb10u1 [1,244 kB]
Fetched 1,244 kB in 0s (12.6 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
dpkg: warning: downgrading sudo from 1.8.27-1+deb10u2 to 1.8.27-1+deb10u1
(Reading database ... 48062 files and directories currently installed.)
Preparing to unpack .../sudo_1.8.27-1+deb10u1_amd64.deb ...
Unpacking sudo (1.8.27-1+deb10u1) over (1.8.27-1+deb10u2) ...
Setting up sudo (1.8.27-1+deb10u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.
Processing triggers for systemd (241-7~deb10u3) ...
root@osestaging1-discourse-ose:/var/www/discourse# dpkg -l | grep -i sudo
ii  sudo                            1.8.27-1+deb10u1             amd64        Provide limited super user privileges to specific users
root@osestaging1-discourse-ose:/var/www/discourse# 



  1. I confirmed cron is running


root@osestaging1-discourse-ose:/var/www/discourse# ps -ef | grep -i cron
root       723   717  0 13:01 ?        00:00:00 runsv cron
root       728   723  0 13:01 ?        00:00:00 cron -f
root      1309   763  0 13:08 pts/1    00:00:00 grep -i cron
root@osestaging1-discourse-ose:/var/www/discourse# 



  1. And our unattended-upgrades cron is in-place


root@osestaging1-discourse-ose:/var/www/discourse# cat /etc/cron.d/unattended-upgrades 
################################################################################
# File:    /etc/cron.d/unattended-upgrades
# Version: 0.1
# Purpose: run unattended-upgrades in lieu of systemd. For more info see
#           * https://wiki.opensourceecology.org/wiki/Discourse
#           * https://meta.discourse.org/t/does-discourse-container-use-unattended-upgrades/136296/3
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2020-03-23
# Updated: 2020-03-23
################################################################################
20 04 * * * root /usr/bin/nice /usr/bin/unattended-upgrades --debug




root@osestaging1-discourse-ose:/var/www/discourse# 



  1. And our patched /etc/pam.d/cron module is fixed


root@osestaging1-discourse-ose:/var/www/discourse# cat /etc/pam.d/cron
# The PAM configuration file for the cron daemon

@include common-auth

# Sets the loginuid process attribute
#session    required     pam_loginuid.so

# Read environment variables from pam_env's default files, /etc/environment
# and /etc/security/pam_env.conf.
session       required   pam_env.so

# In addition, read system locale information
session       required   pam_env.so envfile=/etc/default/locale

@include common-account
@include common-session-noninteractive 

# Sets up user limits, please define limits for cron tasks
# through /etc/security/limits.conf
session    required   pam_limits.so

root@osestaging1-discourse-ose:/var/www/discourse# 



  1. And now I'll wait to see if if my cron job finally actually kicks-off a run of `unattended-upgrades` to upgrade the insecure version of `sudo` within 24 hours!


  1. ...


  1. Meanwhile, back to varnish
    2. 

  2. All of the varnish config is done *outside* discourse. When I left off, I found some 403 errors after switching to varnish (maybe not related). Anyway, let's deal with those mod_security rules
    3. 

  3. Mod_security is setup in the "inner" nginx inside on the Discourse docker container. Here's the current config


root@osestaging1-discourse-ose:/etc/nginx/conf.d# cat modsecurity.include 
################################################################################
# File:    modsecurity.include
# Version: 0.1
# Purpose: Defines mod_security rules for the discourse vhost
#          This should be included in the server{} blocks nginx vhosts.
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2019-11-12
# Updated: 2019-11-12
################################################################################
Include "/etc/modsecurity/modsecurity.conf"

# OWASP Core Rule Set, installed from the 'modsecurity-crs' package in debian
Include /etc/modsecurity/crs/crs-setup.conf
Include /usr/share/modsecurity-crs/rules/*.conf

SecRuleRemoveById 949110, 942360
root@osestaging1-discourse-ose:/etc/nginx/conf.d# 



  1. I also noticed a "414 Request-URI Too Large" response from a GET my browser made to https://discourse.opensourceecology.org/admin/reports/bulk?reports%5Bdau_by_mau%5D%5Bcache%5D=true&reports%5Bdau_by_mau%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bdau_by_mau%5D%5Bstart_date%5D=2020-03-21T00%3A00%3A00.000Z&reports%5Bdau_by_mau%5D%5Bend_date%5D=2020-04-21T23%3A59%3A59.999Z&reports%5Bdaily_engaged_users%5D%5Bcache%5D=true&reports%5Bdaily_engaged_users%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bdaily_engaged_users%5D%5Bstart_date%5D=2020-03-21T00%3A00%3A00.000Z&reports%5Bdaily_engaged_users%5D%5Bend_date%5D=2020-04-21T23%3A59%3A59.999Z&reports%5Bnew_contributors%5D%5Bcache%5D=true&reports%5Bnew_contributors%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bnew_contributors%5D%5Bstart_date%5D=2020-03-21T00%3A00%3A00.000Z&reports%5Bnew_contributors%5D%5Bend_date%5D=2020-04-21T23%3A59%3A59.999Z&reports%5Bpage_view_total_reqs%5D%5Bcache%5D=true&reports%5Bpage_view_total_reqs%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bpage_view_total_reqs%5D%5Bstart_date%5D=2020-03-22T00%3A00%3A00.000Z&reports%5Bpage_view_total_reqs%5D%5Bend_date%5D=2020-04-22T23%3A59%3A59.999Z

    1. fucking hell no wonder. look at that fucking query string. that's insane!
      2. 

    2. but that's only half of it; here's a request cookie '_forum_session'


RmZiVjJySDY1NjVvcXE4QnYwdUpIVVRBYkF2c3JqMGw4WTdGeUhEV1NpakpVQzBRTFIyQnhrN3poUzd6YWIxZ1VHSUQ2S09HSXdnaG9ibUZTZkZkVHhjaThIdk9aK0lhUjBwS2dMd1poa29FMDFqS1pPSUFwM0hyOXF6VW5kY0pyZ21VM1QrbmViNjU2LzY5N0xkcTZ3Qy80OHcvZXZKODkvOGlFVjNKbzZHajZGMmlobU1NYktwZlNuQVh6R3A2UnF1L3hzYWpnTzRGY3JKUFE2SzJJNU55K1ZpSHVWVVJtNWJDT1BodkNRc1NROWtQcVR6S251SlZiSTMvaUsrZjEyRTlqbW9xK0NxN092bkxzeGQxejRlUDR5OTFNOW9SN0YzdUpLMVBvU0hQb1BBV1RKdUtFai9xR05lN2t4dU9SWkQ2U2ZpQmEwcXBDaXFPajdPbGdWOThKOFhaT0VQOWlLTzR4VnJ4KzEwWG5aUis1a3JkOEZQQjl5aHJFTUh2cDB4dm5FYTcyR2IzVkVxMjBSRGh5NnFqUWRsMkcyKzN6Y3krQXZQQUpKZHNCR2JGUWYzUE4rbmhaMUFDZmZGRS0tYTZCSmNDRis4WTRpcmJ2aVQ4d3BxQT09--7d65f5962920fd5f668973ee58e468fd9339f62c



  1. I remember I brought this up to the devs, and they agreed it was silly but didn't commit to fixing it https://meta.discourse.org/t/discourse-session-cookies-400-request-header-or-cookie-too-large/137245

    1. maybe that's a different issue that I already fixed, but this is a consequence of similarlly inconsiderate development practices..
    2. 

  2. so it looks like the option in nginx that would trigger a 414 error is 'large_client_header_buffers'

    1. https://stackoverflow.com/questions/1067334/how-to-set-the-allowed-url-length-for-a-nginx-request-error-code-414-uri-too
      2. 

    2. https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers
    3. 

  3. indeed, as I pointed out in the meta.discourse.org topic above, I've set this as a DOS protection tuning. It runs fine for all our other webapps, but apparently Discourse likes to do stupid shit like using ridiculous large URIs in a GET instead of a POST...


[root@osestaging1 nginx]# hostname
osestaging1
[root@osestaging1 nginx]# pwd
/etc/nginx
[root@osestaging1 nginx]# grep -ir 'large_client_header_buffers' *
nginx.conf:   	;
[root@osestaging1 nginx]# 



  1. apparently the default is "large_client_header_buffers 4 8k", so I set that in the server block of the nginx config specific to the discourse site (this is the "outer" nginx config running on the docker host)


[root@osestaging1 nginx]# grep -ir 'large_client_header_buffers' *
conf.d/discourse.opensourceecology.org.conf:   large_client_header_buffers 4 8k;
nginx.conf:   large_client_header_buffers 2 1k;
[root@osestaging1 nginx]# grep -irC5 'large_client_header_buffers' *
conf.d/discourse.opensourceecology.org.conf-
conf.d/discourse.opensourceecology.org.conf-   # resetting this back to its nginx default to override our DOS protection
conf.d/discourse.opensourceecology.org.conf-   # since the Discourse developers like to store a ton of data on the URI and
conf.d/discourse.opensourceecology.org.conf-   # directly in client's cookies instead of using POST and server-side storage
conf.d/discourse.opensourceecology.org.conf-   #  * https://meta.discourse.org/t/discourse-session-cookies-400-request-header-or-cookie-too-large/137245/6
conf.d/discourse.opensourceecology.org.conf:   large_client_header_buffers 4 8k;
conf.d/discourse.opensourceecology.org.conf-
conf.d/discourse.opensourceecology.org.conf-   # we can't use the global 'secure.include' file for Discourse, which
conf.d/discourse.opensourceecology.org.conf-   # requires use of the DELETE http method, for example
conf.d/discourse.opensourceecology.org.conf-   #include conf.d/secure.include;
conf.d/discourse.opensourceecology.org.conf-
--
nginx.conf-
nginx.conf-   # prevent buffer overflows
nginx.conf-   #client_body_buffer_size  1k;
nginx.conf-   client_body_buffer_size  900k;
nginx.conf-   client_header_buffer_size 1k;
nginx.conf:   large_client_header_buffers 2 1k;
nginx.conf-
nginx.conf-     # allow large posts for image uploads
nginx.conf-     #client_max_body_size 1k;
nginx.conf-     #client_max_body_size 900k;
nginx.conf-     client_max_body_size 10M;
[root@osestaging1 nginx]# 



  1. that's it. I was able to login. logout. login again. edit some settings. create a new topic. No mod_security issues or otherwise.
    2. 

  2. I updated the documentation to use these new nginx config options
    3. 

  3. I did a quick check with `varnishstat` to see if varnish is actually caching. At first the number of hits (all the rows of avg(n) in the top-right) remained all zero. But it was because I was logged-in!
    4. 

  4. I logged-out, deleted my session cookies, and started clicking around the topics and I saw the hit rate climb to ~0.4 (per 10 requests). Unfortunately it quickly dropped back to zero (after maybe 30-120 seconds) after DIscourse set a session cookie, even without logging-in! Ugh, it's a pretty weak cache, but at least it'll do some of the heavy lifting to prevent our site from going down in reddit hug-of-death thundering herd viral event..


Sat Apr 11, 2020


  1. Meeting with Robert


Sun Apr 05, 2020


  1. Researching Robert Hrvol as my potential replacement as OSE Sysadmin
    2. 

  2. I added some items to our OSE Server TODO list https://wiki.opensourceecology.org/wiki/OSE_Server#TODO
    3. 

  3. emailed Marcin about a potential meeting with us 3 to discuss this transition and begin ramping-up Robert and assigning his first tasks


Fri Apr 03, 2020


  1. Our monthly backup report came in with issues:

    1. the first-of-the-month backup is missing for April 1st
      2. 

    2. the day before's March 31st's backup is also missing
      3. 

    3. the day before that's March 30th's backup is actually present in the file listing, but it says that it thinks the day before yesterday's backup is actually April 1st, not March 30th--this one may be a bug in the backup report script




ATTENTION: BACKUPS MISSING!


WARNING: First of this month's backup (20200401) is missing!
WARNING: Yesterday's backup (20200402) is missing!
WARNING: The day before yesterday's backup (20200401) is missing!

See below for the contents of the backblaze b2 bucket = ose-server-backups

monthly_hetzner2_20190501_072001.tar.gpg
monthly_hetzner2_20190601_072001.tar.gpg
monthly_hetzner2_20190701_072001.tar.gpg
monthly_hetzner2_20190901_072001.tar.gpg
monthly_hetzner2_20191001_072001.tar.gpg
monthly_hetzner2_20191101_072001.tar.gpg
monthly_hetzner2_20191201_072001.tar.gpg
monthly_hetzner2_20200201_072001.tar.gpg
monthly_hetzner2_20200301_072001.tar.gpg
weekly_hetzner2_20200316_072001.tar.gpg
weekly_hetzner2_20200323_072001.tar.gpg
weekly_hetzner2_20200330_072001.tar.gpg
yearly_hetzner2_20190101_111520.tar.gpg
yearly_hetzner2_20200101_072001.tar.gpg
---
Note: This report was generated on 20200403_042001 UTC by script '/root/backups/backupReport.sh'
	  This script was triggered by '/etc/cron.d/backup_to_backblaze'

	  For more information about OSE backups, please see the relevant documentation pages on the wiki:
	   * https://wiki.opensourceecology.org/wiki/Backblaze
	   * https://wiki.opensourceecology.org/wiki/OSE_Server#Backup[[https://wiki.opensourceecology.org/wiki/OSE_Server#Backups|s]]









Retrieved from "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log/2020_Q2&oldid=218659"