Maltfield Log

Timesheet

Tue Jun 27, 2017

1. confirmed that 06/26 backups were successfully rsync'd to dreamhost from hetzner1 & hetzner2.
2. the cleanup cron was still in NOP mode, as unlink was commented-out. manually ran cleanup as /home was 99% full. It's now 93% (257G avail) after manually running the cleanup script. I'll need to validate that this cron is fully working later this week.
3. cleanup script on dreamhost leaves behind empty dirs. updated script to clean out empty dirs too.
4. fixed LOCK TABLE permissions issue on wiki mysqldump. Solution was to add the '--single-transaction' argument to the mysqldump command. The file is now 169M after bz2 compression. That's bigger than all the other DBs combined, but still not unreasonably large.
5. meeting with Catarina to generate an ssh keypair
   1. added ssh key to her authorized_keys file & set permissions. validated access successfully.
   2. set her password on the server & validated sudo permissions were already in-place.
   3. fixed permissions on 3DModels directory, set to apache:apache from root recursively
   4. added 'define('ALLOW_UNFILTERED_UPLOADS', true);' to obi's wp-config.php. Because of this, we should not allow untrusted people accounts on this wp site. Currently it's just marcin, me, & cmota. I also asked Catarina to limit files stored on wp to <10mb

Sat Jun 24, 2017

1. Discovered that at least 9.6G out of the 51G on hetzner1 are log files. This should be manually cleaned for now. Hetzner2 should be configured with logrotate, where we delete the files after a few days (now that we'll have backups we don't need to retain logs since 2001.
   1. 7.5G /usr/home/osemain/www_logs
   2. 2.1G /usr/www/users/osemain/logs
   3. Discovered what appears to be a 9.4G (deprecated) backup at /usr/www/users/osemain/w.tar.gz
   4. Discovered another 9.6G (deprecated) backup at /usr/home/osemain/tmpd/upgrade/w.tar.gz
2. Therefore, if I move these files off to /var/tmp/, it will cut the backup size of hetzner1 in less than half.
3. backup cron failed to run on hetzener 2. /var/log/cron says '(time) ERROR (getpwnam() failed)'. I was mising the user between the times & the command (which is unnecessary in hetzner 1, as she uses crontab). Added 'root' as the 7th option before the command, and confirmed it works on non-07:20 test. Logging works as well.
4. backup cron on hetzner 1 ran successfully, though it failed to log. Interesting to note, the filenames are timestamped to 05:20, even though the job is set to run at 07:20. This is because the system time is CEST (UTC+2), but the backup script explicitly generates the timestamp with the -u argument for UTC. This is ideal, as the time of the backup files are unambigious.
5. finished configuring ssh-ident & secpanel for ssh key compartimentalization
6. discovered that dreamhost does have a policy on "unlimited storage", which explicitly says you can't run a file-hosting site, and it seems that using it as a backup archive may violate their "unlimited" policy. We should not depend on dreamhost to not delete out data.. Ideally, we'd need a service that'd give us ~3TB.
7. added cleanLocal.pl to marcin_ose@hancock.dreamhost.com:'/home/marcin_ose/bin/'
8. created crontabs for daily deletion of hetzner1 & hetzner2 backup files from dreamhost that are 3 days old. Note that cleanLocal.pl will intentionally *not* delete any backups created on the 1st day of every month. These will have to be manually deleted every year or so if space becomes an issue. Hopefully I'll have all the backups in perfect shape by July 1st (1 week from today)

Fri Jun 23, 2017

1. Enabling google analytics in cloudflare was wildly successful. We now have the entire day of 06-22 showing:
   1. 1,230 sessions. 1,086 users. 3,024 page views. 3,321 unique page views.
   2. 36% from US, 7% from India, 4% from Canada, then (in order, including all with >=1%): Germany, Australia, UK, Brazil, Neterlands, France, Phillippines, Spain, Italy, Malaysia, Poland, Chile, South Korea, South Africa.
   3. 76% desktop. 20% mobile. 4% tablet.
   4. 7% of page views are '/', 3% "/wiki/List_of_CAD_Programs", 3% '/gvcs', 3% '/gvcs/gvcs-machine-index/', 2% '/wiki/Cost_of_Living', 2% '/wiki/Main_Page', 2% '/wiki/Global_Village_Construction_Set'
   5. the biggest site 45% of traffic
      1. it's especially interesting that Cost_of_Living is more viewed than the wiki home. This page is the 14th result on DDG for 'average cost of living' (which gave us 2 hits yesterday from this search term), and it doesn't even show up on the first 10 pages of google.
   6. 61% of sessions come from a search engine. The top 2 search terms (making up 96%) were not provided.
   7. Referreal traffic is 13% of overall traffic, domains in-order are: youtube.com, pintrest.com, duckduckgo.com, com.google.android.googlequicksearchbox, facebook.com, mg.mail.yahoo.com, ecosia.org, reddit.com, waldenlabs.com
   8. yesterday, there were <50 users at: 12am, 4am, 5am, 6am, 7am, 2pm, 6pm, 7pm, 8pm, 9pm, 10pm, & 11pm. The lowest time was 33 users at 9pm. To avoid ambiguity, I changed the GA UI to UTC, but the hours didn't change--so who knows what this data means! I could also find no information on what "49 users at 1am" means. Is that 01:00-01:59? 00:01-01:00? 00:30-01:29? This wasn't hugely helpful, but it's reasonable to assume that the 00:00-04:00 US PT time is a low-traffic window (as most of our users geolocate to North America). Therefore, I'll start the backups with cron at 07:20 UTC.
2. Hetzner 2 is set to use CEST, which is UTC+2. I'm not sure, but it may switch to CET = UTC+1 sometimes. FeF is UTC-6, but sometimes UTC-5. I'm currently in NYC, which is UTC-4, but sometimes UTC-5. But the OSE devs could be in any timezone. To avoid ambiguity, and ensure consistancy across logs, I'll be changing everything to UTC in the future.
3. Added '/etc/cron.d/backup_to_dreamhost' to hetzner2 to kick-off a backup job at 07:20 UTC (at least it _will be_ UTC in the future; I'm not going to make that change [or any changes] until I've validated automated backups are working with no intervention for at least a few days in a row). Logs go to '/var/log/backups/backup.log'
4. Added a line to hetzner 1's osemain crontab to initiate a backup at 07:20 (this will probably always be CEST). Logs go to /usr/home/osemain/backups/log/backup.log'

Thr Jun 22, 2017

1. First successful execution of the backup.sh script on both servers without manual intervention with all nice & bandwith-caps in-place on both hetzner 1 &2
   1. hetzner 1's mysqldump of the wiki is still failing, requiring an unlock of a table & further research into the potential impact of the change
   2. hetzner1's full backup execution time (including cleaning old local backups, mysqldump, tarball creation, and rsync to dreamhost) is 11 hours. If this server weren't to be deprecated shortly, I'd switch to gz to reduce this, but as space is a concern & the server is temporary, this should suffice.
   3. hetzner2's full backup execution time is under 3 hours

Wed Jun 21, 2017

1. confirmed access to dreamhost web ui
2. confirmed that we have ssh key control from the dreamhost dashboard
3. determined that 'opensourceecology.org' is purchased from dreamhost for $14/yr
   1. found subdomains blog, community, eerik, & forum
   2. found 10 databases on dreamhost: dp7civicrm (drupal 7 civicrm db), dp7crm (drupal 7 civicrm), oftblog (Blog), oftcivi (CiviCRM), oftdrupal, oftforum, oftjoomla, oftsurvey (For LimeSurvey), oftwiki (Wiki), openfarmtech_org (openfarmtech.org/osefriends). Are any in use & in need of backup? Will confirm with Marcin.
   3. confirmed that dreamhost does *not* offer us free https certs (other than letsencrypt.org); they're $15/yr through Comodo. We just go straight with letsencrypt.org
   4. found 5x users on dreamhost: marcin_ose (17G), ose_site (32G), ose_community (0.2G), osecolby (<0.1G), osebackup (<0.1G). The creds I've been given were for marcin_ose, and I'm putting the backups in marcin_ose@hancock.dreamhost.com:/home/marcin_ose/backups/{hetzner1,hetzner2}/$timestamp/
4. The backup of the data on hetzner 1 finished after 7 hours with the following sizes (note it was all bz2 compressed):
   1. 22G public_html (uncompressed size is 31G)
   2. 17G $HOME (uncompressed size is 20G)
   3. 43M mysqldump-forum
   4. 2.4M mysqldump-osemain
   5. 1.2M mysqldump-openswh
   6. 125K mysqldump-fef
   7. 527 bytes mysqldump-wiki
5. There was an issue encountered with the wiki db. I've found a solution command, but I need to research its side-effects in relation to mediawiki to ensure I don't cause any issues
6. 23% space savings probably isn't worth the 7 hour slamming of the CPU to compress at bz2 levels, but I'll keep it this way on hetzner 1, as we've already exceeded our disk quota many times over. I'll switch to gz compression on hetzner 2.
   1. mysqldump: Got error: 1044: Access denied for user 'osewiki_w'@'%' to database 'osewiki' when using LOCK TABLES
7. tx of the 38G from hetzner 1 to dreamhost took just under 1 hour with speeds between 2.82 MB/s - 15.31 MB/s
   1. I'll configure the automated rysnc to cap at 3 MB/s so it takes a reasonable 3-5 hours while reducing the risk of saturating the network bandwidth. TODO: determine the best 5-hour window when the box is most idle on a daily cycle.
8. Added backup scripts to hetzner 2. It does a single root mysqldump + file backups of /etc/, /home/, /var/log/, /root/, and /var/www/. Unlike hetzner 1, we have ample disk space (113G available after the first backup), so I'm using gzip instead of bz2. This whole backup process took 21 min on hetzner2, producing 31G. Transferring this to dreamhost took another 31 min
9. confirmed access to cloudflare acount
   1. their site's "analytics" app has an error & returns an empty data set in both firefox & chrome
   2. it appears that we do have caching enabled, and some other last-mile optimization for mobile
      1. hetzner 2 has 61G of unused RAM. We have plenty of RAM to run a fat cache. Reverse proxy or application-level proxy? Squid or nginx? First, we'll have to see what the bottlenecks are & what the page requests and static vs dynamic content looks like over a few weeks.
   3. WAF events show that 2-200-ish IPs are actively being blocked every day. The worst are attacks on 'wp-login'. This is something we can have ossec or a wp plugin for rate limiting handle with iptables for free
   4. Email Address Obfuscation is currently enabled. There is no good replacement for this afaik, besides author's awareness
   5. "Always on" is enabled. There is no reasonably cheap alternative to this, though it's not strictly a requirement.
10. I added google analytics tracking through cloudflare temporarily so I can get an understanding of the hourly & weekly usage trends of the sites. Eventually I hope to deprecate google analytics for awstats

Tue Jun 20, 2017

1. Determined only backups done on hetzner 1 is mediawiki using MediaWiki's built-in 'maintenance/dumpBackup.php' script to 'w/export/'
2. Confirmed access to mysql databases for mediawiki, main ose wp site, open warehouse wp, fef wp, & oseforum vanilla on old server
3. begun a keepass db as central location for safely storing OSE credentials
4. sent email to Marcin, Tom, and Catarina to get their ssh public keys to populate their user's authorized_keys file prior to disabling password-based authentication
5. added maltfield user to wheel for sudo access
6. discovered that the only 2 dns entries on ghandi are: 'oswarehouse.org' & 'opensourcewarehouse.org'. oswarehouse.org is an unconfigured landing page, and opensourcewarehouse.org points to dreamhost a NS.
7. confirmed ssh access to dreamhost. we're using 18G in our $HOME, and '/home' (which is likley shared with other customers) has 357G free. I'll be using this unlimited storage plan to store compressed tarballs of daily backups of the server's DBs, webroots, and important config file dirs (/etc/, /home/, /root/, etc)
8. was unable to connect to dreamhost web console, messaged Marcin for proper credentials
9. determined size of $HOME on hetzner 1 to be 20G, and size of its 'public_html/' dir to be 31G
10. added backup scripts for mysqldumps of 5x DBs + all files in $HOME (except the backups themselves, of course) to '$HOME/backups/'
11. I noticed that the maximum disk usage for hezner 1's opensourceecology.org domain (not sure how these arbitrary divisions are made between "domains" since everything is actually just thrown into the same '$HOME/public_html' dir--and where would '$HOME' fit anyway?) is 10G, but we're currently using 56G. Trying to create a backup has suspiciously caused my session to be terminated, and when I reconnected I found my screen session no longer existed, suggesting that my whole session was `kill`ed. Using `nice` produced less lethal repercussions.

Mon Jun 19, 2017

• Document as you go along

1. Meeting with Marcin for knowledge transfer on credentials to servers & prioritizing steps to migrate off old server.
2. Established secure channel for credentials exchange
3. Still missing root db credentials & root access to old server Hetzner 1
4. Created a 'maltfield' account on the server, added my ssh public key, and Enabled PubKeyAuthentication on sshd
5. Gained access to opensourcecology.org Google Apps email account with Google Analytics access. I should use this for OSE-related user accounts going forward.

Sun Jun 11, 2017

1. researching free/discounted cloud and/or hardware services for non-profits
2. researching FOSS slack alternatives for real time chatting. decided wire is best.
3. researched bug tracking vs issue tracking software. Found FOSS Request Tracker, OTRS, Liberum Help Desk, GLPI, and Faveo. Marcin mentioned investigations into Mantis

Fri Jun 9, 2017

• Backup Hetzner Old - Wiki, Wordpress, Opensourcewarehouse, Forum, Factor e Farm Blog
• Fix OSE Wordpress (Main Site)
• Move all to Hetzner New. New Hetzner has Openbuildinginstitute. OSE Server
• Install HTTPS
• Install Jitsi Videobridge
• Do dev work
• Discuss password management for IT Team
• Discuss OSE Website Wordpress Theme

Sun Jun 04, 2017

1. Added my video to the FreeCAD_101#Self-Verifying_FreeCAD_Exam_Videos article
2. Added my comment to disqus

Thr June 01, 2017

1. Updated OSE Wiki with better instructions to install the Assembly 2 Workbench FreeCAD Assembly Workbench
2. successfully finished sketching polylines w/ constraints to pocket my 2d initials into the 3d xyz cube
3. recorded myself building another xyz cube, initials, pocketed. Sped-up video to 30-seconds, added soundtrack, & uploaded my finished freecad test to youtube]
4. Created a new repo on my github or my OSE work, and committed/pushed my xyz cube work to here
   1. See video.fcstd for the freecad file that was created during the video's recording.

Tue May 30, 2017

1. Finished MarthamEngineering's 3-part freecad youtube introduction series
2. Began [Marcin's 2-part freecad tutorial]
3. Updated OSE Wiki with better instructions to install Fastener's Workbench Fasteners Workbench in FreeCAD
4. Successfully finished building my first xyz cube in FreeCAD

Mon May 29, 2017

1. Began reading freecad documentation, watching videos on, & playing with freecad
2. Began MarthamEngineering's FreeCAD Tutorials youtube video series, starting with [part 1]

Sat Apr 22, 2017

Online_conferencing