My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

Sat Jan 20, 2018

the db backup & file backup of the wiki finished in a little over an hour

...
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$ time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}
...
real    71m2.031s
user    17m36.700s
sys     1m38.868s
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$

I initiated an scp of the data to hetnzer2

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

transfer of hetzner1 backups to hetzner2 finished after 8 minutes
added wiki.opensourceecology.org A record in cloudflare DNS to 138.201.84.243
disabled CDN/DNS proxy in cloudflare for the 'network' subdomain
decreased TTL of all DNS entries to lowest time possible = 2 minutes for all entries except 'www', 'opensourceecology.org', and 'blog' (all 3 use the CDN for now, and are thus required to be set to TTL = 'automatic')
created/updated necessary config files for forum.opensourceecology.org
1. /etc/httpd/conf.d/00-wiki.opensourceecology.org.conf
2. /etc/varnish/sites-enabled/wiki.opensourceecology.org
3. /etc/varnish/all-vhosts.vcl
4. /etc/nginx/conf.d/wiki.opensourceecology.org.conf
created necessary dirs
1. /var /www/html/wiki.opensourceecology.org/htdocs
2. /var/log/httpd/wiki.opensourceecology.org
3. /var/log/nginx/wiki.opensourceecology.org
updated /etc/php.ini to include "/var/www/html/forum.opensourceecology.org/" in /etc/php.in's "open_basedir"
added wiki SAN to opensourceecology.org cert

certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d stagingforum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org
/bin/chmod 0400 /etc/letsencrypt/archive/*/pri*
nginx -t && service nginx reload

created osewiki_db mysql database on hetzner2 from dump from hetzner1
copied htdocs files on hetzner2 from backup on hetzner1
updated LocalSettings.php with new db name & credentials
updated permissions of docroot files
commented-out $wgScriptPath & $wgStylePath to remove the "/w/" subdir in LocalSettings.php
removed the "/w/" subdir prefix from $wgLogo in LocalSettings.php
commented-out $wgServer & $wgFavicon to prevent a 301 redirect back to the naked domain opensourceecology.org
commented-out $wgCacheDirectory & $wgUseFileCache. And `rm -rf cache`. This directory is *supposed* to be located outside the docroot for security reasons. But we won't need this feature as we use varnish https://www.mediawiki.org/wiki/Manual:Security#File_permissions
added "Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php" to /etc/httpd/conf.d/00-wiki.opensourceecology.org
commented-out debugging lines in LocalSettings.php

################################################################                                                                                            
# Debugging                                                                                                                                                 
# error_reporting(E_ALL | E_STRICT);                                                                                                                        
# error_reporting(E_ALL);                                                                                                                                   
# ini_set("display_errors", 1);                                                                                                                             
																																							
# $wgShowExceptionDetails = true; ## Verbose output to user                                                                                                 
																																							
#$wgShowSQLErrors = true;                                                                                                                                   
#$wgDebugDumpSql  = true;                                                                                                                                   
																																							
#$wgDebugLogFile = "/usr/home/osemain/public_html/logs/wiki-error.log";                                                                                     
#$wgDebugLogFile = "/home/oft_site/logs/wiki-error.log";                                                                                                    
# $wgDebugRawPage = true;                                                                                                                                   
# $wgDebugComments = true;                                                                                                                                  
################################################################

saw some errors for generating temporary thumbnails

Error creating thumbnail: Unable to save thumbnail to destination

1. attempted setting the image directory to be writeable, to no avail

find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;

began investigating the guide to install Mediawiki via git https://www.mediawiki.org/wiki/Download_from_Git

time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git

pushd ${backupDir_hetzner2}/current

determined that the latest version of Mediawiki is v1.30.0, and that we're currently running v1.24.2 https://wiki.opensourceecology.org/wiki/Special:Version

Thr Jan 18, 2018

fixed awstats cron job & config files
confirmed that a db dump includes image tags with the domain-name hard-coded in the href. that was put in-place by Wordpress's "Add Media" wui button. That's not good; the links should be relative in the db!
1. did some research & found that wp core devs decided 7 years ago to keep absolute paths. This is especially bad for continuous integration or even a basic staging site

* https://core.trac.wordpress.org/ticket/17048

there's no good, robust solution.

* https://stackoverflow.com/questions/17187437/relative-urls-in-wordpress
* https://wordpress.org/plugins/relative-url/
* https://wordpress.org/plugins/root-relative-urls/
* https://deluxeblogtips.com/relative-urls/
* http://www.456bereastreet.com/archive/201010/how_to_make_wordpress_urls_root_relative/

take away is:
1. Let wordpress do its thing. Don't waste effort fighting wp when it auto-inserts an absolute path.
2. However, whenever you have to manually type a path in (ie: when configuring a widget, plugin nav bar, etc), please use a relative link.
attempted to fix the "Http error" reported by wordpress after attempting to upload a large image
1. using the browser debugger, I saw that it was nginx that returned a 413 error. I fixed this by increasing 'client_max_body_size' to '10M' in /etc/nginx/nginx.conf

[root@hetzner2 dbChange.20180118_12:22:36]# grep -B 1 'client_max_body_size' /etc/nginx/nginx.conf
		# allow large posts for image uploads
		#client_max_body_size 1k;
		#client_max_body_size 900k;
		client_max_body_size 10M;

1. next, I got a 403 error from /wp-admin/async-upload.php
  1. /var/log/httpd/fef.opensourceecology.org/error_log shows a modsecurity issue:

{code}

> /var/log/httpd/fef.opensourceecology.org/error_log <

[Thu Jan 18 14:56:25.263164 2018] [:error] [pid 27682] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "219"] [id "960915"] [rev "1"] [msg "Multipart parser detected a possible unmatched boundary."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "fef.opensourceecology.org"] [uri "/wp-admin/async-upload.php"] [unique_id "WmC1mU7FUiUY6HdrzSWfWgAAAAA"] {code}

1. 1. as above, whitelisted rule IDs:
    1. 960915, multipart_unmatched_boundary
    2. 200003, multipart_unmatched_boundary
moved '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org' to '/usr/home/osemain/noBackup/deleteMeIn2019/oswh_olddocroot'
added a 301 redirect from 'http://opensourcewarehouse.org' to 'https://oswh.opensourceecology.org' in new file = '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org/index.php'

Sat Jan 13, 2018

Meeting with Marcin

Fri Jan 12, 2018

finished configuring oswh wp plugins
determined that the oswh hack was responsible for injecting pop-ups for a "windows defender" phishing site on some subset of page loads
gained access to oseforums, oseblog, osecivi users via ssh on our non-dedicated server (hetzner1)
checked /etc/passwd & found another 8x org-specific users with home directories that I couldn't access still
emailed hetzner for advice on how to gain ssh access to these users' home directories

addon:x:1011:1011:addontest.opensourceecology.org:/usr/home/addon:bin/false
oseirc:x:1018:1018:irc.opensourceecology.org:/usr/home/oseirc:bin/false
oseholla:x:1019:1019:holla.opensourceecology.org:/usr/home/oseholla:bin/false
osesurv:x:1020:1020:survey.opensourceecology.org:/usr/home/osesurv:/bin/bash
sandbox:x:1021:1021:sandbox.opensourceecology.org:/usr/home/sandbox:/bin/false
microft:x:1022:1022:test.opensourceecology.org:/usr/home/microft:/bin/bash
zabbix:x:118:121::/var/run/zabbix:/bin/false
openswh:x:1023:1023:opensourcewarehouse.org:/usr/home/openswh:/bin/false

created backup of oseforum's docroot (58M) & db (44M). Both sizes are bzip2'd.

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
 dbUser_hetzner2="oseforum_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

Mon Jan 08, 2018

tried installing the fresh version of the Eventor theme to the website running the old wp core, but it was still broken
reverted to the docroot I backed-up before attempting to try the outdated wp core && installed _that_ Eventor theme to the fresh version 1.7 that was just downloaded, and the site actually worked!
finally, attempted the wp-cli commands to update themes & plugins && install our minimal set of sec plugins. The site was still functional after

Fri Jan 05, 2018

investigation of minor fef issues

Thr Jan 04, 2018

downloaded the Eventor theme v 1.7, thanks to Simone's contact with Themes Kingdom
Hetzner responded saying we can use WebFTP to uplaod to $HOME by clicking "the server at the top"
Marcin responded with some issues with osemain's ephemeral clone
Catarina found some linking issues in fef
1. I brought the site down & did a string replacement for all occurrences of 'http://opensourceecology.org/fef' to '/', brought the site back up, and asked Catarina to check again
2. updated documentation at Wordpress#replace_strings_everywhere_in_wp_database_backend

Wed Jan 03, 2018

migrated fef to hetzner2 CHG-2018-01-03
updated statuscake for obi to hit 'https://www.openbuildinginstitute.org'
updated statuscake for fef to hit 'https://fef.opensourceecology.org'
ensured that ssh was activated for all domains/users on our (apparently dedicated, per hetzner support) hetzner1 server (but without root access) via the konsoleh site -> click on the server -> Account Management -> SSH access -> Select domain (for each) -> Next
the kosoleh wui only allowed editing files in the docroot, not the user's home-dir, which prevented me from actually adding my ssh pubic key to $HOME/.ssh/authorized_keys file
I emailed hetzner support back asking if [a] they could just add my pub key to all our user account's authorized_keys files or [b] tell me how I could reset all the user's passwords
oswh was cannibalized by a virus & is awaiting a fresh version of the theme. the forums is awaiting access to the user account. I'm now going to work on beginning the migration of osemain
1. it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
  1. archive
  2. w
  3. logs
  4. mediawiki-1.24.2.extra
2. the entire dir is 23G. Excluding the above, it's ~ 0.7G

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
 dbUser_hetzner1="${mysqlUser_osemain}"
 dbPass_hetzner1="${mysqlPass_osemain}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}

1. the gz-compressed tarball generated from above was 353M.

# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180103"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
 dbUser_hetzner2="osemain_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

created domain name 'osemain.opensourceecology.org' for testing the osemain site on hetzner2
using above vars, I followed the guide to migrate the files & db data from hetzner1 to hetzner2 Wordpress#migrate_site_from_hetzner1_to_hetzner2
created necessary files & dirs:
1. /etc/httpd/conf.d/00-www.opensourceecology.org.conf
2. /etc/varnish/sites-enabled/www.opensourceecology.org
3. /etc/nginx/conf.d/www.opensourceecology.org.conf
  1. this file has a temporary override for the 'Host' header passed to varnish, since the staging url is going to be 'osemain.opensourceecology.org' but the prod site will be 'opensourceecology.org'
4. /var/log/httpd/www.opensourceecology.org
5. /var/log/nginx/www.opensourceecology.org
updated necessary files
1. /etc/varnish/all-vhosts.vcl
2. /etc/php.ini
finished setting up ephemeral clone of osemain at https://osemain.opensourceecology.org
1. sent email to Marcin & Catarina for validation

Tue Jan 02, 2018

got an email from Simone Cicero stating that she emailed Themes Kingdom for a clean copy of Eventor 1.7
emailed back-and-forth with hetzner
1. learned that the forums are in /usr/www/users/oseforum/
2. learned that we have a bunch of users on this box, and it might even be dedicated just for us (though without root access)

osemain@dedi978:~$ grep 'ose' /etc/group
users:x:100:osemain,addon,osecivi,oseblog,oseforum,oseirc,oseholla,osesurv,sandbox,microft,openswh
osemain:x:1010:
osecivi:x:1014:
oseblog:x:1015:
oseforum:x:1016:
oseirc:x:1018:
oseholla:x:1019:
osesurv:x:1020:

1. but I couldn't actually access the home dirs of the other users through 'osemain'

osemain@dedi978:~$ date
Tue Jan  2 16:21:13 CET 2018
osemain@dedi978:~$ ls -lah /usr/home/
ls: cannot open directory /usr/home/: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/addon
ls: cannot open directory /usr/home/addon: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osecivi
ls: cannot open directory /usr/home/osecivi: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseblog
ls: cannot open directory /usr/home/oseblog: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseirc
ls: cannot open directory /usr/home/oseirc: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseforum
ls: cannot open directory /usr/home/oseforum: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osesurv
ls: cannot open directory /usr/home/osesurv: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/openswh
ls: cannot open directory /usr/home/openswh: Permission denied

1. so I asked hetzner support to add the 'osemain' user to all the other users groups listed above, and I asked them to find any other accounts that we own that I may have missed

Maltfield Log/2018 Q1

Contents

See Also

Sat Jan 20, 2018

Thr Jan 18, 2018

> /var/log/httpd/fef.opensourceecology.org/error_log <

Sat Jan 13, 2018

Fri Jan 12, 2018

Mon Jan 08, 2018

Fri Jan 05, 2018

Thr Jan 04, 2018

Wed Jan 03, 2018

Tue Jan 02, 2018

Navigation menu

Maltfield Log/2018 Q1

See Also

Sat Jan 20, 2018

Thr Jan 18, 2018

> /var/log/httpd/fef.opensourceecology.org/error_log <

Sat Jan 13, 2018

Fri Jan 12, 2018

Mon Jan 08, 2018

Fri Jan 05, 2018

Thr Jan 04, 2018

Wed Jan 03, 2018

Tue Jan 02, 2018

Navigation menu

Search