Maltfield Log

From Open Source Ecology
Revision as of 02:50, 12 August 2017 by Maltfield (talk | contribs)
Jump to navigation Jump to search

Links: Timesheet, OSE_Server

Fri, Aug 11, 2017

  1. commented-out monitoring of kern.log (I got nearly 1,000 ossec alerts since enabling them almost a week ago
  2. added /var/log/kern.log, which was 42M since its first entry less than a month ago
  3. updated common log format for apache to track sessionid & time to generate
  4. confirmed that ossec is now reporting diffs in alerts
  5. installed mod_evasive
  6. confirmed that mod_evasive is running & actively blocking DOS attacks (or at least attempting to)
  7. disabled webdav by commenting out all lines in /etc/httpd/conf.modules.d/00-dav.conf
  8. commented-out mod_info in /etc/httpd/conf.modules.d/00-base.conf
  9. moved mod_security overrides to the vhost file (/etc/httpd/conf.d/obi.conf) by Location match (where needed)
  10. removed file /etc/httpd/modsecurity.d/modsecurity_crs_00_config.conf
  11. finished apache hardening (minus changes needed for https & related hardening)
  12. added documentation on mod_security to the OSE_Server page
  13. mysql (mariadb) hardneing
    1. added 'skip-networking', 'skip-show-database', and 'local-infile=0' to /etc/my.cnf
    2. reset root password to a good, long, random passphrase (and updated keepass)
    3. updated /root/backups/backup.settings with the new mysql root password
    4. dropped unnecessary users: osemain, osewiki, & osewiki_w & unnecssary hosts '127.0.0.1' & '::1' & 'centos-72-64-minimal'. Only 2 entries remain: 'root'@'localhost' & 'wordpressuser'@'localhost'
  14. began hardening obi
    1. defined all salts in wp-config.php. Passwords in the db will only be updated when a user logs-in again. We'll force this to happen by resetting everyone's password once https is implemented.
    2. set DISALLOW_FILE_EDIT in wp-config.php
    3. renamed wordpress db & created new user
  15. researched ~10 2FA/OTP wordpress plugins
    1. best option is probably 'google-authenticator' https://wordpress.org/plugins/google-authenticator
      1. this doesn't support forcing users to use 2FA, but this can be achieved with this additional plugin https://wordpress.org/plugins/google-authenticator-encourage-user-activation/
oldDbName=wordpress
newDbName=obi_db
oldUsername=wordpressuser
newUser=obi_user
newPass=CHANGEME
 rootDbPass=CHANGEME

sudo su -
stamp=`date +%Y%m%d_%T`
tmpDir=/var/tmp/dbChange.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
service httpd stop

# create backup
time nice mysqldump -uroot -p --all-databases | gzip -c > preBackup.all_databases.sql.gz

# create new db
echo "CREATE DATABASE $newDbName; GRANT ALL PRIVILEGES ON $newDbName.* TO '$newUser'@'localhost' IDENTIFIED BY '$newPass'; FLUSH PRIVILEGES;" | mysql -uroot -p$rootDbPass

# move tables from old db to new db
 mysql -uroot -p$rootDbPass $oldDbName -sNe 'show tables' | while read table; do mysql -uroot -p$rootDbPass -sNe "RENAME TABLE $oldDbName.$table TO $newDbName.$table;"; done

# update the wp-config.php file & start httpd
    1. verified that the site was working, then deleted the old user
DROP USER 'wordpressuser'@'localhost';

Tue, Aug 08, 2017

  1. keepass training meeting with Marcin & Christian

Sat, Aug 05, 2017

  1. added marcin to recieve monthly statuscake reports
  2. isolated the cookie obi issue to the "Secure" flag, so I enabled it to be just "HttpOnly" to provide some XSS protection, until we can enable the original line after implementing https
  3. installed git on hetzner2
  4. installed mod_security_crs
  5. added '/etc/httpd/modesecurity.d/modsecurity_crs_00_config.conf' to disable several rules, which were triggering false positives.
  6. added ossec local rule to calm down alerts of mod_security
  7. added /var/log/cron to ossec
  8. added /var/log/kern.log to ossec
  9. added ossec local rule to ignore alerts on kern.log for IN=eth0. We should get an alert if something tries to get out, but attempts to get in are unavoidably continuious
    1. also ignored OUT=lo; not sure why this is happening, but I don't think it's necessary
  10. added report_changes="yes" and realtime="yes" to the directories listed undersyscheck in /var/ossec/etc/ossec.conf
  11. updated log_format of mariadb logfile to 'mysql_log'

Fri, Aug 04, 2017

  1. confirmed that hetzner1's daily backups dropped from 39G to 14G starting on 2017-08-03 due to the cleanup from 08-02
  2. hardened keepas file
    1. now has a randomly generated 100-character password
    2. also requires the 4 KiB key file
    3. increased transformation rounds to 87654321, which takes about 5 seconds on my 2-year-old laptop
    4. moved to /etc/keepass, which is owned by root:keepass 770. The files inside are root:keepass 660.
  3. emailed Marcin & Christian for keepass hand-off/training session

Thr, Aug 03, 2017

  1. generated a 4 KiB key file for passwords keepass file
    1. discovered that dd will take an undeterministic size from /dev/random, regardless of bs & count. used head instead
    2. `head -c 4096 < /dev/random > ose.passwords.key`

Wed, Aug 02, 2017

  1. confirmed the hetzner1 2017-08-01 backup of 39G is on backup server
  2. created /usr/home/osemain/noBackup on hetzner1
  3. added "--exclude /usr/home/osemain/noBackup" to backup.sh on hetzner1 to reduce nightly backup size from 39G to 12G
  4. fixed wp-login.php cookie issue preventing logins on obi
    1. ERROR: Cookies are blocked or not supported by your browser. You must enable cookies to use WordPress
    2. fix was commenting-out "Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure" in /etc/httpd/conf/httpd.conf

Tue, Aug 01, 2017

  1. meeting with Lex
  2. sent Marcin my signed Liability Waiver for Aug 25 workshop

Mon Jul 31, 2017

  1. moved obi's wp-config.php up 1 dir to be outside the docroot & chmod to 400
  2. begun debugging oib wp-login issues (cookie error)
    1. confirmed that I cannot login
    2. did not see any error logs
    3. began reverting all session-related hardening changes to php.ini

Sat Jul 29, 2017

  1. multi-user keepass research

Fri Jul 28, 2017

  1. added christian to statuscake admin contact group
  2. emailed christian & marcin about gpg key creation for recieving ossec alerts

Thr Jul 27, 2017

  1. email to Devs about new github accounts for forking filament winder
  2. created account for christian on hetzner2

Thr Jul 20, 2017

  1. further docker research
  2. dialog with Lex about docker
  3. meeting with Christian about sysadmin assistance

Wed Jul 19, 2017

  1. keepass research
  2. preparing mediawiki file dump for Lex
  3. container research

Tue Jul 18, 2017

  1. emailed with Marcin about containers
  2. emailed Lex asking for his ssh public key
    1. created user on hetzner 2
    2. added user to sshaccess
    3. added Lex's new public key
    4. uploaded last night's dump of the wiki to /var/tmp/ with a symlink in his $HOME per Marcin's request
  3. confirmed that "fake" backup "from" 2017-07-01 of hetzner1 was not deleted, so it's safe to make changes to hetzner1 now
  4. moved 27G of assumed-unnecessary files from hetzner1 into '/usr/home/osemain/deleteMeIn2018/'
    1. if nothing is reported as broken by 2017-08-02, I'll validate that the files were included in the first-of-the-month August backup, then add an "exclude" argument to the tarball creation in 'backup.sh' for 'deleteMeIn2018'. And I will not be migrating these files to hetzner2.
  5. finishing php hardening
    1. changed upload_tmp_dir to '/var/lib/php/tmp_upload' per best practices
    2. added upload_tmp_dir to basedir
    3. set session.use_strict_mode = 1
    4. set session.cookie_httponly = 1
    5. changed PHPSESSID to custom value (session.name)
    6. changed session.save_path to "/var/lib/php/session"
    7. set the session.hash_function to 'sha512'
    8. disable_functions = ini_set,php_uname,getmyuid,getmypid,passthru,leak,listen,diskfreespace,tmpfile,link,ignore_user_abord,shell_exec,dl,set_time_limit,exec,system,highlight_file,source,show_source,fpaththru,virtual,posix_ctermid,posix_getcwd,posix_getegid,posix_geteuid,posix_getgid,posix_getgrgid,posix_getgrnam,posix_getgroups,posix_getlogin,posix_getpgid,posix_getpgrp,posix_getpid,posix,_getppid,posix_getpwnam,posix_getpwuid,posix_getrlimit,posix_getsid,posix_getuid,posix_isatty,posix_kill,posix_mkfifo,posix_setegid,posix_seteuid,posix_setgid,posix_setpgid,posix_setsid,posix_setuid,posix_times,posix_ttyname,posix_uname,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,phpinfo,popen,curl_exec,curl_multi_exec,parse_ini_file,allow_url_fopen,allow_url_include,pcntl_exec,chgrp,chmod,chown,lchgrp,lchown,putenv
    9. soap.wsdl_cache_dir = /var/lib/php/soap_cache
  6. began hardening apache
    1. reset permissions of files:
      1. chown -R apache:apache /var/www/html
      2. find /var/www/html/ -type d -exec chmod 750 {} \;
      3. find /var/www/html/ -type f -exec chmod 640 {} \;
    2. added ServerTokens Prod
    3. added ServerSignature Off
    4. removed DocumentRoot from the main /etc/httpd/conf/httpd.conf since we're using vhosts
      1. removed cooresponding Directory block
    5. added 'Options -Indexes -Includes' to all Directory blocks
    6. added "Order allow,deny/nAllow from all" to all Directory blocks
    7. set main "Directory /" block to only contain "Options -Indexes -Includes\nAllowOverride none"
    8. added 'FileETag None' option
    9. added "TraceEnable off" option
    10. recursively changed ownership of all files in /var/www/html/obi/htdocs to 'apache:apache'
      1. changed wp-config.php back to 'root:root'
    11. recursively changed ownership of '/etc/httpd/conf' & '/etc/httpd/conf.d' to apache:apache & set permissions to 750 (from root:root & 755)
      1. recursively changed permissions to 640 within the above dirs
    12. added LimitExcept block to deny all requests other than the 3 basic: GET, POST, & HEAD for all vhost Directory blocks
    13. added "Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure" for XSS protection
    14. added "Header always append X-Frame-Options SAMEORIGIN" for clickjacking protection
    15. added mod_rewrite rules to disable HTTP 1.0 to avoid session hijacking risks
    16. decreased apache timeout to 60 (default is 300) to decrease DoS risk
    17. installed mod_security

Mon Jul 17, 2017

  1. validated with Marcin that obi's content saves are now functioning, following my php.ini changes from yesterday

Sun Jul 16, 2017

  1. Recieved an email from marcin describing issues adding content to OBI wordpress
  2. grepped through /var/log/error_log & found issues with php temp dir & max_input_vars due to recent php hardening that may be breaking wordpress changes
    1. relevant errors:
      1. [Sun Jul 16 04:34:18.532015 2017] [:error] [pid 21569] [client 184.157.59.85:45406] PHP Warning: Unknown: Input variables exceeded 100. To increase the limit change max_input_vars in php.ini. in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4509&action=edit
      2. [Sun Jul 16 07:11:02.922720 2017] [:error] [pid 24088] [client 184.157.59.85:49642] PHP Warning: File upload error - unable to create a temporary file in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4531&action=edit
      3. [Sun Jul 16 07:11:35.723209 2017] [:error] [pid 23510] [client 184.157.59.85:49646] PHP Warning: Unknown: open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/var/www/html/obi/:/var/www/html/osemain) in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4531&action=edit
    2. changed upload_tmp_dir to '/var/lib/php/tmp_upload' per best practices (currently it's '/tmp')
    3. set the new tmp_upload dir to root:apache 770, unlike '/tmp' which is necessarily 777
    4. increased max_input_vars back to 1000, the default
  3. emailed with marcin about configuring his mail client with gpg for ossec reports
  4. statuscake started marking oswh as "recovered" at 04:08 ET, after a 9-hour false-negative streak. I confirmed that they have not responded to my support question about the issue.

Sat Jul 15, 2017

  1. email exchange with marcin about security limitations
  2. investigating statuscake's false-negative "outage" of opensourcewarehouse.org
    1. confirmed that 200 OK was returned with `curl -I "http://www.opensourcewarehouse.org/"`
    2. sent statuscake a support message about the false-negative

Fri Jul 14, 2017

  1. Got Christian's ssh pubkey & asked for availability for a meeting
  2. Got credentials for themeforest, but was unable to login due to a shitty 2FA over email
  3. Asked Catarina to login to themeforest & send me the osemain theme's "Item Purchase Code" so that I can register an account with the theme creator's forum https://support.livemeshthemes.com/wp-login.php?action=register
  4. gained access to themeforest & downloaded all necessary files
  5. fixed osemain wp theme issue
    1. I downloaded the theme archive from themeforest
    2. I scp'd the archive to /usr/home/osemain/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme.zip
    3. I extracted the archive & the archive within named enigmatic.zip to /usr/home/osemain/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme/enigmatic
    4. I changed directory to /usr/home/osemain/public_html/wp-content/themes
    5. I changed the current directory & the 'enigmatic' directory's permissions to '755'
    6. `mv enigmatic enigmatic.20170714.bak` && `rsync -av --progress ~/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme/enigmatic .`
    7. I emailed Marcin for validation of the changes
  6. created a user on osemain for Tom

Thr Jul 13, 2017

  1. fixed migration oversite that broke all the obi pages but the main page
    1. new docroot was missing /var/log/http/obi/htdocs/.htaccess, which included mod_rewrite rules
  2. emailed Christian asking for portfolio & CV for risk analysis before user creation
  3. manually added an admin user for myself to the osemain wp db (ose_website) as uid = 55 using SQL commands directly
  4. login attempt produced a fatal php error:
    1. Fatal error: Call to undefined function ot_register_meta_box() in /usr/www/users/osemain/wp-content/themes/enigmatic/framework/presentation/metabox-manager.php on line 34
  5. created new user 'osemain' on mysql db of hetzner2 & database 'osemain' per the wordpress install guide
  6. copied last night's backup of the osemain db's mysqldump sql file to hetzner2 from backup server
  7. copied public_html from backup server to hetzner2 & extracted to /var/www/html/osemain.old/htdocs/
  8. created vhost at /etc/httpd/conf.d/osemain.conf
  9. manually reset theem through mysql to 'twentyeleven'
  10. successfully logged-in to osemain on the new server under new theme
    1. confirmed wp is currently 4.7.5 on osemain

Tue Jul 11, 2017

  1. verified that the cleanup of files on hetzner2 reduced the daily backup size from 20G to 1.3G
    1. The dir 'ose_wiki' (16G) & file w.tar.gz (9.4G) had been moved to /var/tmp/deleteMeIn2018/ a few days ago
  2. confirmed that ossec has been actively blocking abusive ip addresses
    1. <active-response> is configured to block these ip addresses via host-deny & iptables for 10 minutes
    2. logs are found in /var/ossec/logs/active-responses.log
    3. I was able to ban myself by failing ssh attempt 10 times in a row
  3. organized httpd vhost as prereq to harden php
    1. created /etc/httpd/conf.d/obi.conf
    2. moved all files from /var/www/html to /var/www/html/obi/htdocs/
      1. soon obi-related files that should be stored outside the docroot for security reasons can be stored in the 'obi' dir, but the docroot will be 'htdocs' ie: wp-config.php!
  4. hardened php
    1. disabled allow_url_fopen
    2. reduced max_input_time & max_execution_time to 30 seconds
    3. disabled expose_php
    4. decreased max_input_vars to 100
    5. decreased post_max_size & upload_max_filesize to 10M
    6. added open_basedir to whitelist the php dir "/var/www/html/obi/"
  5. researched our https options with multiple domain names running as vhosts on a single server
    1. sent an emal to Marcin & Catarina asking if they'd be interested in consolidating to a single domain with many subdomains to simplify the costs & config
    2. discovered that Hetzner charges € 0.84 / month for each additional ip address, which is our best non-consolidated option, as there's client support issues with SNI & running on separate ports is unpractical
    3. research shows that ~10% of internet users do *not* support SNI as of last year (2016)
    4. Catarina said she wants separate domains, and she said she'd cover the 10 eur/yr fee

Mon Jul 10, 2017

  1. installed make, gcc, gcc-c++, kernel-devs
  2. installed & started ossec service
    1. added config to monitor mariadb logs
  3. installed & started postfix service
  4. added TMOUT=14400 to '/root/.bash_profile' to force all root logins to timeout after 4 hours
  5. configured ossec to send pgp-encrypted emails
    1. installed procmail & mailx
    2. configured mailbox_command in /etc/postfix/main.cf
    3. imported my gpg public key to keyring @ /var/ossec/.gnupg/
    4. created /var/ossec/.procmailrc
    5. created script /var/ossec/sent_encrypted_alarm.sh
  6. emailed Catarina & gained admin access to OBI WP
  7. receieved many OSSEC alerts regarding brute-force attempts on wp-login.php from a distributed set of IPs
    1. researched wp-login.php hardning options, including 2FA, captcha, rate limiting plugins, and ossec active response iptables ip address blocking. I'm thinking we'll do all except the captcha, especially if 2FA can be enforced for all users.

Sun Jul 09, 2017

  1. discovered that hezner2 has 2x 250G disks: sda & sdb
    1. each disk has 3 partitions of exactly the same size, and is RAID1'd between their cooresponding partition numbers on each disk
      1. 1 = 34.4G
      2. 2 = 537M
      3. 215G
    2. RAID config is as follows:
      1. /dev/md0 = 34.3G = swap
      2. /dev/md1 = 536M = ext3. mounted at '/boot'. filesystem features includes has_journal.
      3. /dev/md2 = 215G = ext4. mounted at '/'. `df` total size is listed as 197G. flesystem features includes has_journal.
    3. currently, 83G are in-use out of 197G
  2. reset the root password on hetzner2. added new credential to keepass.
  3. updated all packages (210) with `yum update`
  4. hardened sshd_config
    1. set 'PermitRootLogin no'
    2. set 'PasswordAuthentication no'
    3. set 'PermitEmptyPasswords no'
    4. set 'IgnoreRhosts yes'
    5. set 'AllowGroups sshaccess'
      1. created group 'sshaccess' & added all 4 users
  5. added documentation to OSE_Server on how to add new users with ssh access to the server
  6. created new, hardened ssh host keys
    1. `ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -b 4096 -o -a 100 -N `
    2. `ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -t ecdsa -b 521 -o -a 100 -N `
    3. `ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -t ed25519 -a 100 -N `
  7. moved some large, seemingly unnecessary files from /var/www/html/ to /var/tmp/deleteMeIn2018/ to reduce backup sizes
  8. updated logrotate config to decrease backup sizes
    1. added 'compress' & 'delaycompress' to /etc/logrotate.conf
    2. changed default rotation from 'weekly' to 'daily' & rotate from '4' to '32' to keep the same amount of data locally, but now compressed daily (per above)
    3. added /etc/logrotate.d/backups for our new backup log
  9. updated the hostname of hetzner2 to 'hetzner2.opensourceecology.org' in /etc/hosts, /etc/hostname, and using `hostnamectl`
  10. set timezone to UTC with `timedatectl set-timezone UTC`
  11. restarted rsyslog & confirmed hostname & date updates got pushed to /var/log/secure
  12. replaced firewalld with iptables
  13. configured iptables
    1. apache & mysql users cannot send traffic out unless it's already established or dns
    2. no traffic can come in over eth0 unless it's over ssh or http or established

Sat Jul 08, 2017

  1. researching GPL & BSD license info
  2. got an email from statuscake that the site was down, verified that hitting opensourceecology.org in my browser lead tme to the cloudflare error page. so confirmed that statuscake works for email alerts & outages even when using cloud flare's "always on" cache
  3. added status.opensourceecology.org CNAME to statuscake.com
  4. added open building institue to statuscake
  5. added documentation to OSE Server about statuscake
  6. emailed OSE Devs, providing them with the statuscake URL & asking if anything else should be monitored that I missed
  7. created fake backup '20170701' from June 5th's backup as a long-term backup that won't be deleted by the cleanup script (the 1st of every month is normally kept, but Hetzner brought our site down, so backups were missing from the 1st through 3rd) by setting the mtime of the files with 'find . -type f | xargs touch -d '20170701'
  8. Added 'keepass solution' to TODO on OSE Server

Fri Jul 07, 2017

  1. researching OSE wiki's contents for Licensing info

Thr Jul 06, 2017

  1. created email statuscake at opensourceecology.org & added credentials to keepass
  2. created statuscake & added crednetials to keepass
  3. meeting with marcin, who now has a password-protected ssh keypair that is verified to work with both ssh & filezilla (sftp)
  4. configured (untested) statuscake basic GET checks with a public reporting page https://uptime.statuscake.com/?TestID=itmHX7Pfj2

Tue Jul 04, 2017

  1. SSH is now working to Hetzner 1, and I validated that backups are running again
    1. The last backup before the outage that still existed on dreamhost was from 2017-06-30. I made a copy of this outside the 'backups/hetzner1/' dir in $HOME so it's preserved (not deleted by the cron) just in case
    2. The first backup we have after the outage is from today, 2017-07-04. This started automatically, indicating that the site was offline at 07:20 for the daily backup for 3 days in a row. That's pretty awful service from Hetzner!
  2. Verified that my email address was added to the admin/techincal emails list on the hetzner1 web console
  3. Submitted a support request asking for [1] an RCA of the outage they caused [2] a way to communicate with Hetzner that has confidentiality [3] access to "the Robot"
  4. Added Tom's ssh key to hetzner1's authorized_keys file
  5. Emailed Marcin how I can create a new Gapps Email account as prereq to signing up for statuscake

Mon Jul 03, 2017

  1. Site came back online with no word from hetzner, but we still cannot ssh in.
  2. Hetzner support is still unresponsive to my numerous emails
  3. Spent some time researching hetzner alternatives that use 100% renewable energy & offer dedicated servers for comparable prices

Sun Jul 02, 2017

  1. The site was still down this morning, and hetzner 1 was still inaccessible. No word from Marcin
    1. I sent an email to support@heztner.de asking if our server was offline, indicating that we lost it about the time of their maintenance window
    2. I logged into cloudflare, and saw that opensourceecology.org points to '78.46.3.178'. Note that this is distinct from `dig` results, as the cloudflare service is essentially a MITM
      1. could not ping or ssh into 78.46.3.178
      2. determined that our old 'dedi978.your-server.de` dns host resolves to the above-listed IP address. That whold endpoint is insaccessible. I hope that we can find a static IP address that's associated with our vServer, and reconfigure to use that directly.

Sat Jul 01, 2017

  1. 07/01 backups from hetzner #2 were successful
  2. 07/01 backups from hetzner #1 failed
    1. I checked if they were still rsync-ing, and was unable to ssh into the server
    2. I saw mail from Marcin indicating the server was out
    3. I could not ping our server on dedi978.your-server.de'
    4. I could not preform any admin actions from the hetzner web portal
    5. Google analtyics still showed traffic to our site, perhaps a false negative due to the cloudflare "always on" feature
    6. I found that hetzner recently did maintenance to their VHosts, indicating that our vServer's vHost may change, and that we should check "robot" to determine our new vHost https://www.hetzner-status.de/en.html
    7. I found the URL for the "robot" service, but was unable to login with the credentials given to me by Marcin https://robot.your-server.de/
    8. I messaged Marcin, asking if he had these credentials

Thr Jun 29, 2017

  1. confirmed that 2017-06-29 backups came in, though hetzner1's dropped to 33G--not sure why

Wed Jun 28, 2017

  1. watched Monday's meeting
  2. Added my profile to the OSE Developers wiki article, including avatar & badge upload
  3. Added email & website support to the Department template Template:Department
  4. Met with Marcin
  5. Documented the backup procedure on OSE Server

Tue Jun 27, 2017

  1. confirmed that 06/26 backups were successfully rsync'd to dreamhost from hetzner1 & hetzner2.
  2. the cleanup cron was still in NOP mode, as unlink was commented-out. manually ran cleanup as /home was 99% full. It's now 93% (257G avail) after manually running the cleanup script. I'll need to validate that this cron is fully working later this week.
  3. cleanup script on dreamhost leaves behind empty dirs. updated script to clean out empty dirs too.
  4. fixed LOCK TABLE permissions issue on wiki mysqldump. Solution was to add the '--single-transaction' argument to the mysqldump command. The file is now 169M after bz2 compression. That's bigger than all the other DBs combined, but still not unreasonably large.
  5. meeting with Catarina to generate an ssh keypair
    1. added ssh key to her authorized_keys file & set permissions. validated access successfully.
    2. set her password on the server & validated sudo permissions were already in-place.
    3. fixed permissions on 3DModels directory, set to apache:apache from root recursively
    4. added 'define('ALLOW_UNFILTERED_UPLOADS', true);' to obi's wp-config.php. Because of this, we should not allow untrusted people accounts on this wp site. Currently it's just marcin, me, & cmota. I also asked Catarina to limit files stored on wp to <10mb

Sat Jun 24, 2017

  1. Discovered that at least 9.6G out of the 51G on hetzner1 are log files. This should be manually cleaned for now. Hetzner2 should be configured with logrotate, where we delete the files after a few days (now that we'll have backups we don't need to retain logs since 2001.
    1. 7.5G /usr/home/osemain/www_logs
    2. 2.1G /usr/www/users/osemain/logs
    3. Discovered what appears to be a 9.4G (deprecated) backup at /usr/www/users/osemain/w.tar.gz
    4. Discovered another 9.6G (deprecated) backup at /usr/home/osemain/tmpd/upgrade/w.tar.gz
  2. Therefore, if I move these files off to /var/tmp/, it will cut the backup size of hetzner1 in less than half.
  3. backup cron failed to run on hetzener 2. /var/log/cron says '(time) ERROR (getpwnam() failed)'. I was mising the user between the times & the command (which is unnecessary in hetzner 1, as she uses crontab). Added 'root' as the 7th option before the command, and confirmed it works on non-07:20 test. Logging works as well.
  4. backup cron on hetzner 1 ran successfully, though it failed to log. Interesting to note, the filenames are timestamped to 05:20, even though the job is set to run at 07:20. This is because the system time is CEST (UTC+2), but the backup script explicitly generates the timestamp with the -u argument for UTC. This is ideal, as the time of the backup files are unambigious.
  5. finished configuring ssh-ident & secpanel for ssh key compartimentalization
  6. discovered that dreamhost does have a policy on "unlimited storage", which explicitly says you can't run a file-hosting site, and it seems that using it as a backup archive may violate their "unlimited" policy. We should not depend on dreamhost to not delete out data.. Ideally, we'd need a service that'd give us ~3TB.
  7. added cleanLocal.pl to marcin_ose@hancock.dreamhost.com:'/home/marcin_ose/bin/'
  8. created crontabs for daily deletion of hetzner1 & hetzner2 backup files from dreamhost that are 3 days old. Note that cleanLocal.pl will intentionally *not* delete any backups created on the 1st day of every month. These will have to be manually deleted every year or so if space becomes an issue. Hopefully I'll have all the backups in perfect shape by July 1st (1 week from today)

Fri Jun 23, 2017

  1. Enabling google analytics in cloudflare was wildly successful. We now have the entire day of 06-22 showing:
    1. 1,230 sessions. 1,086 users. 3,024 page views. 3,321 unique page views.
    2. 36% from US, 7% from India, 4% from Canada, then (in order, including all with >=1%): Germany, Australia, UK, Brazil, Neterlands, France, Phillippines, Spain, Italy, Malaysia, Poland, Chile, South Korea, South Africa.
    3. 76% desktop. 20% mobile. 4% tablet.
    4. 7% of page views are '/', 3% "/wiki/List_of_CAD_Programs", 3% '/gvcs', 3% '/gvcs/gvcs-machine-index/', 2% '/wiki/Cost_of_Living', 2% '/wiki/Main_Page', 2% '/wiki/Global_Village_Construction_Set'
    5. the biggest site 45% of traffic
      1. it's especially interesting that Cost_of_Living is more viewed than the wiki home. This page is the 14th result on DDG for 'average cost of living' (which gave us 2 hits yesterday from this search term), and it doesn't even show up on the first 10 pages of google.
    6. 61% of sessions come from a search engine. The top 2 search terms (making up 96%) were not provided.
    7. Referreal traffic is 13% of overall traffic, domains in-order are: youtube.com, pintrest.com, duckduckgo.com, com.google.android.googlequicksearchbox, facebook.com, mg.mail.yahoo.com, ecosia.org, reddit.com, waldenlabs.com
    8. yesterday, there were <50 users at: 12am, 4am, 5am, 6am, 7am, 2pm, 6pm, 7pm, 8pm, 9pm, 10pm, & 11pm. The lowest time was 33 users at 9pm. To avoid ambiguity, I changed the GA UI to UTC, but the hours didn't change--so who knows what this data means! I could also find no information on what "49 users at 1am" means. Is that 01:00-01:59? 00:01-01:00? 00:30-01:29? This wasn't hugely helpful, but it's reasonable to assume that the 00:00-04:00 US PT time is a low-traffic window (as most of our users geolocate to North America). Therefore, I'll start the backups with cron at 07:20 UTC.
  2. Hetzner 2 is set to use CEST, which is UTC+2. I'm not sure, but it may switch to CET = UTC+1 sometimes. FeF is UTC-6, but sometimes UTC-5. I'm currently in NYC, which is UTC-4, but sometimes UTC-5. But the OSE devs could be in any timezone. To avoid ambiguity, and ensure consistancy across logs, I'll be changing everything to UTC in the future.
  3. Added '/etc/cron.d/backup_to_dreamhost' to hetzner2 to kick-off a backup job at 07:20 UTC (at least it _will be_ UTC in the future; I'm not going to make that change [or any changes] until I've validated automated backups are working with no intervention for at least a few days in a row). Logs go to '/var/log/backups/backup.log'
  4. Added a line to hetzner 1's osemain crontab to initiate a backup at 07:20 (this will probably always be CEST). Logs go to /usr/home/osemain/backups/log/backup.log'

Thr Jun 22, 2017

  1. First successful execution of the backup.sh script on both servers without manual intervention with all nice & bandwith-caps in-place on both hetzner 1 &2
    1. hetzner 1's mysqldump of the wiki is still failing, requiring an unlock of a table & further research into the potential impact of the change
    2. hetzner1's full backup execution time (including cleaning old local backups, mysqldump, tarball creation, and rsync to dreamhost) is 11 hours. If this server weren't to be deprecated shortly, I'd switch to gz to reduce this, but as space is a concern & the server is temporary, this should suffice.
    3. hetzner2's full backup execution time is under 3 hours

Wed Jun 21, 2017

  1. confirmed access to dreamhost web ui
  2. confirmed that we have ssh key control from the dreamhost dashboard
  3. determined that 'opensourceecology.org' is purchased from dreamhost for $14/yr
    1. found subdomains blog, community, eerik, & forum
    2. found 10 databases on dreamhost: dp7civicrm (drupal 7 civicrm db), dp7crm (drupal 7 civicrm), oftblog (Blog), oftcivi (CiviCRM), oftdrupal, oftforum, oftjoomla, oftsurvey (For LimeSurvey), oftwiki (Wiki), openfarmtech_org (openfarmtech.org/osefriends). Are any in use & in need of backup? Will confirm with Marcin.
    3. confirmed that dreamhost does *not* offer us free https certs (other than letsencrypt.org); they're $15/yr through Comodo. We just go straight with letsencrypt.org
    4. found 5x users on dreamhost: marcin_ose (17G), ose_site (32G), ose_community (0.2G), osecolby (<0.1G), osebackup (<0.1G). The creds I've been given were for marcin_ose, and I'm putting the backups in marcin_ose@hancock.dreamhost.com:/home/marcin_ose/backups/{hetzner1,hetzner2}/$timestamp/
  4. The backup of the data on hetzner 1 finished after 7 hours with the following sizes (note it was all bz2 compressed):
    1. 22G public_html (uncompressed size is 31G)
    2. 17G $HOME (uncompressed size is 20G)
    3. 43M mysqldump-forum
    4. 2.4M mysqldump-osemain
    5. 1.2M mysqldump-openswh
    6. 125K mysqldump-fef
    7. 527 bytes mysqldump-wiki
  5. There was an issue encountered with the wiki db. I've found a solution command, but I need to research its side-effects in relation to mediawiki to ensure I don't cause any issues
  6. 23% space savings probably isn't worth the 7 hour slamming of the CPU to compress at bz2 levels, but I'll keep it this way on hetzner 1, as we've already exceeded our disk quota many times over. I'll switch to gz compression on hetzner 2.
    1. mysqldump: Got error: 1044: Access denied for user 'osewiki_w'@'%' to database 'osewiki' when using LOCK TABLES
  7. tx of the 38G from hetzner 1 to dreamhost took just under 1 hour with speeds between 2.82 MB/s - 15.31 MB/s
    1. I'll configure the automated rysnc to cap at 3 MB/s so it takes a reasonable 3-5 hours while reducing the risk of saturating the network bandwidth. TODO: determine the best 5-hour window when the box is most idle on a daily cycle.
  8. Added backup scripts to hetzner 2. It does a single root mysqldump + file backups of /etc/, /home/, /var/log/, /root/, and /var/www/. Unlike hetzner 1, we have ample disk space (113G available after the first backup), so I'm using gzip instead of bz2. This whole backup process took 21 min on hetzner2, producing 31G. Transferring this to dreamhost took another 31 min
  9. confirmed access to cloudflare acount
    1. their site's "analytics" app has an error & returns an empty data set in both firefox & chrome
    2. it appears that we do have caching enabled, and some other last-mile optimization for mobile
      1. hetzner 2 has 61G of unused RAM. We have plenty of RAM to run a fat cache. Reverse proxy or application-level proxy? Squid or nginx? First, we'll have to see what the bottlenecks are & what the page requests and static vs dynamic content looks like over a few weeks.
    3. WAF events show that 2-200-ish IPs are actively being blocked every day. The worst are attacks on 'wp-login'. This is something we can have ossec or a wp plugin for rate limiting handle with iptables for free
    4. Email Address Obfuscation is currently enabled. There is no good replacement for this afaik, besides author's awareness
    5. "Always on" is enabled. There is no reasonably cheap alternative to this, though it's not strictly a requirement.
  10. I added google analytics tracking through cloudflare temporarily so I can get an understanding of the hourly & weekly usage trends of the sites. Eventually I hope to deprecate google analytics for awstats

Tue Jun 20, 2017

  1. Determined only backups done on hetzner 1 is mediawiki using MediaWiki's built-in 'maintenance/dumpBackup.php' script to 'w/export/'
  2. Confirmed access to mysql databases for mediawiki, main ose wp site, open warehouse wp, fef wp, & oseforum vanilla on old server
  3. begun a keepass db as central location for safely storing OSE credentials
  4. sent email to Marcin, Tom, and Catarina to get their ssh public keys to populate their user's authorized_keys file prior to disabling password-based authentication
  5. added maltfield user to wheel for sudo access
  6. discovered that the only 2 dns entries on ghandi are: 'oswarehouse.org' & 'opensourcewarehouse.org'. oswarehouse.org is an unconfigured landing page, and opensourcewarehouse.org points to dreamhost a NS.
  7. confirmed ssh access to dreamhost. we're using 18G in our $HOME, and '/home' (which is likley shared with other customers) has 357G free. I'll be using this unlimited storage plan to store compressed tarballs of daily backups of the server's DBs, webroots, and important config file dirs (/etc/, /home/, /root/, etc)
  8. was unable to connect to dreamhost web console, messaged Marcin for proper credentials
  9. determined size of $HOME on hetzner 1 to be 20G, and size of its 'public_html/' dir to be 31G
  10. added backup scripts for mysqldumps of 5x DBs + all files in $HOME (except the backups themselves, of course) to '$HOME/backups/'
  11. I noticed that the maximum disk usage for hezner 1's opensourceecology.org domain (not sure how these arbitrary divisions are made between "domains" since everything is actually just thrown into the same '$HOME/public_html' dir--and where would '$HOME' fit anyway?) is 10G, but we're currently using 56G. Trying to create a backup has suspiciously caused my session to be terminated, and when I reconnected I found my screen session no longer existed, suggesting that my whole session was `kill`ed. Using `nice` produced less lethal repercussions.

Mon Jun 19, 2017

  • Document as you go along
  1. Meeting with Marcin for knowledge transfer on credentials to servers & prioritizing steps to migrate off old server.
  2. Established secure channel for credentials exchange
  3. Still missing root db credentials & root access to old server Hetzner 1
  4. Created a 'maltfield' account on the server, added my ssh public key, and Enabled PubKeyAuthentication on sshd
  5. Gained access to opensourcecology.org Google Apps email account with Google Analytics access. I should use this for OSE-related user accounts going forward.

Sun Jun 11, 2017

  1. researching free/discounted cloud and/or hardware services for non-profits
  2. researching FOSS slack alternatives for real time chatting. decided wire is best.
  3. researched bug tracking vs issue tracking software. Found FOSS Request Tracker, OTRS, Liberum Help Desk, GLPI, and Faveo. Marcin mentioned investigations into Mantis

Fri Jun 9, 2017

  • Backup Hetzner Old - Wiki, Wordpress, Opensourcewarehouse, Forum, Factor e Farm Blog
  • Fix OSE Wordpress (Main Site)
  • Move all to Hetzner New. New Hetzner has Openbuildinginstitute. OSE Server
  • Install HTTPS
  • Install Jitsi Videobridge
  • Do dev work
  • Discuss password management for IT Team
  • Discuss OSE Website Wordpress Theme

Sun Jun 04, 2017

  1. Added my video to the FreeCAD_101#Self-Verifying_FreeCAD_Exam_Videos article
  2. Added my comment to disqus

Thr June 01, 2017

  1. Updated OSE Wiki with better instructions to install the Assembly 2 Workbench FreeCAD Assembly Workbench
  2. successfully finished sketching polylines w/ constraints to pocket my 2d initials into the 3d xyz cube
  3. recorded myself building another xyz cube, initials, pocketed. Sped-up video to 30-seconds, added soundtrack, & uploaded my finished freecad test to youtube]
  4. Created a new repo on my github or my OSE work, and committed/pushed my xyz cube work to here
    1. See video.fcstd for the freecad file that was created during the video's recording.

Tue May 30, 2017

  1. Finished MarthamEngineering's 3-part freecad youtube introduction series
  2. Began [Marcin's 2-part freecad tutorial]
  3. Updated OSE Wiki with better instructions to install Fastener's Workbench Fasteners Workbench in FreeCAD
  4. Successfully finished building my first xyz cube in FreeCAD

Mon May 29, 2017

  1. Began reading freecad documentation, watching videos on, & playing with freecad
  2. Began MarthamEngineering's FreeCAD Tutorials youtube video series, starting with [part 1]

Sat Apr 22, 2017

Online_conferencing

Retrieved from "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log&oldid=157488"