Maltfield Log/2018 Q1

My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

1. Maltfield_Log
2. User:Maltfield
3. Special:Contributions/Maltfield

Fri Jan 05, 2018

1. investigation of minor fef issues

Thr Jan 04, 2018

1. downloaded the Eventor theme v 1.7, thanks to Simone's contact with Themes Kingdom
2. Hetzner responded saying we can use WebFTP to uplaod to $HOME by clicking "the server at the top"
3. Marcin responded with some issues with osemain's ephemeral clone
4. Catarina found some linking issues in fef
   1. I brought the site down & did a string replacement for all occurrences of 'http://opensourceecology.org/fef' to '/', brought the site back up, and asked Catarina to check again
   2. updated documentation at Wordpress#replace_strings_everywhere_in_wp_database_backend

Wed Jan 03, 2018

1. migrated fef to hetzner2 CHG-2018-01-03
2. updated statuscake for obi to hit 'https://www.openbuildinginstitute.org'
3. updated statuscake for fef to hit 'https://fef.opensourceecology.org'
4. ensured that ssh was activated for all domains/users on our (apparently dedicated, per hetzner support) hetzner1 server (but without root access) via the konsoleh site -> click on the server -> Account Management -> SSH access -> Select domain (for each) -> Next
5. the kosoleh wui only allowed editing files in the docroot, not the user's home-dir, which prevented me from actually adding my ssh pubic key to $HOME/.ssh/authorized_keys file
6. I emailed hetzner support back asking if [a] they could just add my pub key to all our user account's authorized_keys files or [b] tell me how I could reset all the user's passwords
7. oswh was cannibalized by a virus & is awaiting a fresh version of the theme. the forums is awaiting access to the user account. I'm now going to work on beginning the migration of osemain
   1. it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
      1. archive
      2. w
      3. logs
      4. mediawiki-1.24.2.extra
   2. the entire dir is 23G. Excluding the above, it's ~ 0.7G

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
 dbUser_hetzner1="${mysqlUser_osemain}"
 dbPass_hetzner1="${mysqlPass_osemain}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}

1. 1. the gz-compressed tarball generated from above was 353M.

# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180103"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
 dbUser_hetzner2="osemain_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

1. created domain name 'osemain.opensourceecology.org' for testing the osemain site on hetzner2
2. using above vars, I followed the guide to migrate the files & db data from hetzner1 to hetzner2 Wordpress#migrate_site_from_hetzner1_to_hetzner2
3. created necessary files & dirs:
   1. /etc/httpd/conf.d/00-www.opensourceecology.org.conf
   2. /etc/varnish/sites-enabled/www.opensourceecology.org
   3. /etc/nginx/conf.d/www.opensourceecology.org.conf
      1. this file has a temporary override for the 'Host' header passed to varnish, since the staging url is going to be 'osemain.opensourceecology.org' but the prod site will be 'opensourceecology.org'
   4. /var/log/httpd/www.opensourceecology.org
   5. /var/log/nginx/www.opensourceecology.org
4. updated necessary files
   1. /etc/varnish/all-vhosts.vcl
   2. /etc/php.ini
5. finished setting up ephemeral clone of osemain at https://osemain.opensourceecology.org
   1. sent email to Marcin & Catarina for validation

Tue Jan 02, 2018

1. got an email from Simone Cicero stating that she emailed Themes Kingdom for a clean copy of Eventor 1.7
2. emailed back-and-forth with hetzner
   1. learned that the forums are in /usr/www/users/oseforum/
   2. learned that we have a bunch of users on this box, and it might even be dedicated just for us (though without root access)

osemain@dedi978:~$ grep 'ose' /etc/group
users:x:100:osemain,addon,osecivi,oseblog,oseforum,oseirc,oseholla,osesurv,sandbox,microft,openswh
osemain:x:1010:
osecivi:x:1014:
oseblog:x:1015:
oseforum:x:1016:
oseirc:x:1018:
oseholla:x:1019:
osesurv:x:1020:

1. 1. but I couldn't actually access the home dirs of the other users through 'osemain'

osemain@dedi978:~$ date
Tue Jan  2 16:21:13 CET 2018
osemain@dedi978:~$ ls -lah /usr/home/
ls: cannot open directory /usr/home/: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/addon
ls: cannot open directory /usr/home/addon: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osecivi
ls: cannot open directory /usr/home/osecivi: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseblog
ls: cannot open directory /usr/home/oseblog: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseirc
ls: cannot open directory /usr/home/oseirc: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseforum
ls: cannot open directory /usr/home/oseforum: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osesurv
ls: cannot open directory /usr/home/osesurv: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/openswh
ls: cannot open directory /usr/home/openswh: Permission denied

1. 1. so I asked hetzner support to add the 'osemain' user to all the other users groups listed above, and I asked them to find any other accounts that we own that I may have missed