Maltfield Log: Difference between revisions

From Open Source Ecology
Jump to navigation Jump to search
No edit summary
No edit summary
 
(118 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[Timesheet]]
{{RightTOC}}
Links: [[OSE_Server]]. [[Michael Photo Folder]].


=Sat Jun 24, 2017=
<html><iframe width="700" height="600" src="https://osedev.org/wiki/Maltfield?start=1/1/2019"></iframe></html>
# Discovered that at least 9.6G out of the 51G on hetzner1 are log files. This should be manually cleaned for now. Hetzner2 should be configured with logrotate, where we delete the files after a few days (now that we'll have backups we don't need to retain logs since 2001.
## 7.5G /usr/home/osemain/www_logs
## 2.1G /usr/www/users/osemain/logs
## Discovered what appears to be a 9.4G (deprecated) backup at /usr/www/users/osemain/w.tar.gz
## Discovered another 9.6G (deprecated) backup at /usr/home/osemain/tmpd/upgrade/w.tar.gz
# Therefore, if I move these files off to /var/tmp/, it will cut the backup size of hetzner1 in less than half.
# backup cron failed to run on hetzener 2. /var/log/cron says '(time) ERROR (getpwnam() failed)'. I was mising the user between the times & the command (which is unnecessary in hetzner 1, as she uses crontab). Added 'root' as the 7th option before the command, and confirmed it works on non-07:20 test. Logging works as well.
# backup cron on hetzner 1 ran successfully, though it failed to log. Interesting to note, the filenames are timestamped to 05:20, even though the job is set to run at 07:20. This is because the system time is CEST (UTC+2), but the backup script explicitly generates the timestamp with the -u argument for UTC. This is ideal, as the time of the backup files are unambigious.
# finished configuring ssh-ident & secpanel for ssh key compartimentalization
# discovered that dreamhost does have a policy on "unlimited storage", which explicitly says you can't run a file-hosting site, and it seems that using it as a backup archive may violate their "unlimited" policy. We should not depend on dreamhost to not delete out data.. Ideally, we'd need a service that'd give us ~3TB.
# added cleanLocal.pl to marcin_ose@hancock.dream[[host.com:'/home/marcin_ose/bin/']]
# created crontabs for daily deletion of hetzner1 & hetzner2 backup files from dreamhost that are 3 days old. Note that cleanLocal.pl will intentionally *not* delete any backups created on the 1st day of every month. These will have to be manually deleted every year or so if space becomes an issue. Hopefully I'll have all the backups in perfect shape by July 1st (1 week from today)


=Fri Jun 23, 2017=
Link: https://osedev.org/wiki/Maltfield
# Enabling google analytics in cloudflare was wildly successful. We now have the entire day of 06-22 showing:
## 1,230 sessions. 1,086 users. 3,024 page views. 3,321 unique page views.
## 36% from US, 7% from India, 4% from Canada, then (in order, including all with >=1%): Germany, Australia, UK, Brazil, Neterlands, France, Phillippines, Spain, Italy, Malaysia, Poland, Chile, South Korea, South Africa.
## 76% desktop. 20% mobile. 4% tablet.
## 7% of page views are '/', 3% "/wiki/List_of_CAD_Programs", 3% '/gvcs', 3% '/gvcs/gvcs-machine-index/', 2% '/wiki/Cost_of_Living', 2% '/wiki/Main_Page', 2% '/wiki/Global_Village_Construction_Set'
## the biggest site 45% of traffic
### it's especially interesting that Cost_of_Living is more viewed than the wiki home. This page is the 14th result on DDG for 'average cost of living' (which gave us 2 hits yesterday from this search term), and it doesn't even show up on the first 10 pages of google.
## 61% of sessions come from a search engine. The top 2 search terms (making up 96%) were not provided.
## Referreal traffic is 13% of overall traffic, domains in-order are: youtube.com, pintrest.com, duckduckgo.com, com.google.android.googlequicksearchbox, facebook.com, mg.mail.yahoo.com, ecosia.org, reddit.com, waldenlabs.com
## yesterday, there were <50 users at: 12am, 4am, 5am, 6am, 7am, 2pm, 6pm, 7pm, 8pm, 9pm, 10pm, & 11pm. The lowest time was 33 users at 9pm. To avoid ambiguity, I changed the GA UI to UTC, but the hours didn't change--so who knows what this data means! I could also find no information on what "49 users at 1am" means. Is that 01:00-01:59? 00:01-01:00? 00:30-01:29? This wasn't hugely helpful, but it's reasonable to assume that the 00:00-04:00 US PT time is a low-traffic window (as most of our users geolocate to North America). Therefore, I'll start the backups with cron at 07:20 UTC.
# Hetzner 2 is set to use CEST, which is UTC+2. I'm not sure, but it may switch to CET = UTC+1 sometimes. FeF is UTC-6, but sometimes UTC-5. I'm currently in NYC, which is UTC-4, but sometimes UTC-5. But the OSE devs could be in any timezone. To avoid ambiguity, and ensure consistancy across logs, I'll be changing everything to UTC in the future.
# Added '/etc/cron.d/backup_to_dreamhost' to hetzner2 to kick-off a backup job at 07:20 UTC (at least it _will be_ UTC in the future; I'm not going to make that change [or any changes] until I've validated automated backups are working with no intervention for at least a few days in a row). Logs go to '/var/log/backups/backup.log'
# Added a line to hetzner 1's osemain crontab to initiate a backup at 07:20 (this will probably always be CEST). Logs go to /usr/home/osemain/backups/log/backup.log'


=Thr Jun 22, 2017=
=Key Wiki Pages=
# First successful execution of the backup.sh script on both servers without manual intervention with all nice & bandwith-caps in-place on both hetzner 1 &2
# [[User:Maltfield]]
## hetzner 1's mysqldump of the wiki is still failing, requiring an unlock of a table & further research into the potential impact of the change
# [[OSE Server]] and [[OSE Server#Backups]]
## hetzner1's full backup execution time (including cleaning old local backups, mysqldump, tarball creation, and rsync to dreamhost) is 11 hours. If this server weren't to be deprecated shortly, I'd switch to gz to reduce this, but as space is a concern & the server is temporary, this should suffice.
# [[Web server configuration]]
## hetzner2's full backup execution time is under 3 hours
# [[Wordpress]]
# [[Mediawiki]]
# [[:Category:CHGs]]


=Wed Jun 21, 2017=
=Logs=
# confirmed access to dreamhost web ui
# confirmed that we have ssh key control from the dreamhost dashboard
# determined that 'opensourceecology.org' is purchased from dreamhost for $14/yr
## found subdomains blog, community, eerik, & forum
## found 10 databases on dreamhost: dp7civicrm (drupal 7 civicrm db), dp7crm (drupal 7 civicrm), oftblog (Blog), oftcivi (CiviCRM), oftdrupal, oftforum, oftjoomla, oftsurvey (For LimeSurvey), oftwiki (Wiki), openfarmtech_org (openfarmtech.org/osefriends). Are any in use & in need of backup? Will confirm with Marcin.
## confirmed that dreamhost does *not* offer us free https certs (other than letsencrypt.org); they're $15/yr through Comodo. We just go straight with letsencrypt.org
## found 5x users on dreamhost: marcin_ose (17G), ose_site (32G), ose_community (0.2G), osecolby (<0.1G), osebackup (<0.1G). The creds I've been given were for marcin_ose, and I'm putting the backups in marcin_ose@hancock.dreamhost.com:/home/marcin_ose/backups/{hetzner1,hetzner2}/$timestamp/
# The backup of the data on hetzner 1 finished after 7 hours with the following sizes (note it was all bz2 compressed):
## 22G public_html (uncompressed size is 31G)
## 17G $HOME (uncompressed size is 20G)
## 43M mysqldump-forum
## 2.4M mysqldump-osemain
## 1.2M mysqldump-openswh
## 125K mysqldump-fef
## 527 bytes mysqldump-wiki
# There was an issue encountered with the wiki db. I've found a solution command, but I need to research its side-effects in relation to mediawiki to ensure I don't cause any issues
# 23% space savings probably isn't worth the 7 hour slamming of the CPU to compress at bz2 levels, but I'll keep it this way on hetzner 1, as we've already exceeded our disk quota many times over. I'll switch to gz compression on hetzner 2.
## mysqldump: Got error: 1044: Access denied for user 'osewiki_w'@'%' to database 'osewiki' when using LOCK TABLES
# tx of the 38G from hetzner 1 to dreamhost took just under 1 hour with speeds between 2.82 MB/s - 15.31 MB/s
## I'll configure the automated rysnc to cap at 3 MB/s so it takes a reasonable 3-5 hours while reducing the risk of saturating the network bandwidth. TODO: determine the best 5-hour window when the box is most idle on a daily cycle.
# Added backup scripts to hetzner 2. It does a single root mysqldump + file backups of /etc/, /home/, /var/log/, /root/, and /var/www/. Unlike hetzner 1, we have ample disk space (113G available after the first backup), so I'm using gzip instead of bz2. This whole backup process took 21 min on hetzner2, producing 31G. Transferring this to dreamhost took another 31 min
# confirmed access to cloudflare acount
## their site's "analytics" app has an error & returns an empty data set in both firefox & chrome
## it appears that we do have caching enabled, and some other last-mile optimization for mobile
### hetzner 2 has 61G of unused RAM. We have plenty of RAM to run a fat cache. Reverse proxy or application-level proxy? Squid or nginx? First, we'll have to see what the bottlenecks are & what the page requests and static vs dynamic content looks like over a few weeks.
## WAF events show that 2-200-ish IPs are actively being blocked every day. The worst are attacks on 'wp-login'. This is something we can have ossec or a wp plugin  for rate limiting handle with iptables for free
## Email Address Obfuscation is currently enabled. There is no good replacement for this afaik, besides author's awareness
## "Always on" is enabled. There is no reasonably cheap alternative to this, though it's not strictly a requirement.
# I added google analytics tracking through cloudflare temporarily so I can get an understanding of the hourly & weekly usage trends of the sites. Eventually I hope to deprecate google analytics for awstats


=Tue Jun 20, 2017=
In early 2018 I discovered that this page had become the [[Special:LongPages|single longest article in our wiki]], so I segregated out my work log by year.
# Determined only backups done on hetzner 1 is mediawiki using MediaWiki's built-in 'maintenance/dumpBackup.php' script to 'w/export/'
# Confirmed access to mysql databases for mediawiki, main ose wp site, open warehouse wp, fef wp, & oseforum vanilla on old server
# begun a keepass db as central location for safely storing OSE credentials
# sent email to Marcin, Tom, and Catarina to get their ssh public keys to populate their user's authorized_keys file prior to disabling password-based authentication
# added maltfield user to wheel for sudo access
# discovered that the only 2 dns entries on ghandi are: 'oswarehouse.org' & 'opensourcewarehouse.org'. oswarehouse.org is an unconfigured landing page, and opensourcewarehouse.org points to dreamhost a NS.
# confirmed ssh access to dreamhost. we're using 18G in our $HOME, and '/home' (which is likley shared with other customers) has 357G free. I'll be using this unlimited storage plan to store compressed tarballs of daily backups of the server's DBs, webroots, and important config file dirs (/etc/, /home/, /root/, etc)
# was unable to connect to dreamhost web console, messaged Marcin for proper credentials
# determined size of $HOME on hetzner 1 to be 20G, and size of its 'public_html/' dir to be 31G
# added backup scripts for mysqldumps of 5x DBs + all files in $HOME (except the backups themselves, of course) to '$HOME/backups/'
# I noticed that the maximum disk usage for hezner 1's opensourceecology.org domain (not sure how these arbitrary divisions are made between "domains" since everything is actually just thrown into the same '$HOME/public_html' dir--and where would '$HOME' fit anyway?) is 10G, but we're currently using 56G. Trying to create a backup has suspiciously caused my session to be terminated, and when I reconnected I found my screen session no longer existed, suggesting that my whole session was `kill`ed. Using `nice` produced less lethal repercussions.


=Mon Jun 19, 2017=
In mid-2018, I discovered that my work log had become so large that it exceeded Mediawiki's allowable limits (triggering a "413 Request Entity Too Large" error), so I segregated out my work log by quarter into subpages.
*Document as you go along
# Meeting with Marcin for knowledge transfer on credentials to servers & prioritizing steps to migrate off old server.
# Established secure channel for credentials exchange
# Still missing root db credentials & root access to old server Hetzner 1
# Created a 'maltfield' account on the server, added my ssh public key, and Enabled PubKeyAuthentication on sshd
# Gained access to opensourcecology.org Google Apps email account with Google Analytics access. I should use this for OSE-related user accounts going forward.


=Sun Jun 11, 2017=
# researching free/discounted cloud and/or hardware services for non-profits
# researching FOSS slack alternatives for real time chatting. decided wire is best.
# researched bug tracking vs issue tracking software. Found FOSS Request Tracker, OTRS, Liberum Help Desk, GLPI, and Faveo. Marcin mentioned investigations into Mantis


=Fri Jun 9, 2017=
# [[Maltfield_Log/2025_Q2|2025 Q2 Logs for Michael Altfield]]
*Backup Hetzner Old - Wiki, Wordpress, Opensourcewarehouse, Forum, Factor e Farm Blog
# [[Maltfield_Log/2025_Q1|2025 Q1 Logs for Michael Altfield]]
*Fix OSE Wordpress (Main Site)
# [[Maltfield_Log/2024_Q4|2024 Q4 Logs for Michael Altfield]]
*Move all to Hetzner New. New Hetzner has Openbuildinginstitute. [[OSE Server]]
# [[Maltfield_Log/2024_Q3|2024 Q3 Logs for Michael Altfield]]
*Install HTTPS
# [[Maltfield_Log/2024_H1|2024 Q1 & Q2 Logs for Michael Altfield]]
*Install Jitsi Videobridge
# [[Maltfield_Log/2023|2023 Logs for Michael Altfield]]
*Do dev work
# [[Maltfield_Log/2022|2022 Logs for Michael Altfield]]
*Discuss password management for IT Team
# [[Maltfield_Log/2021_Q3|2021 Q3 Logs for Michael Altfield]]
*Discuss OSE Website Wordpress Theme
# [[Maltfield_Log/2021_Q2|2021 Q2 Logs for Michael Altfield]]
 
# [[Maltfield_Log/2020_Q3|2020 Q3 Logs for Michael Altfield]]
=Sun Jun 04, 2017=
# [[Maltfield_Log/2020_Q2|2020 Q2 Logs for Michael Altfield]]
# Added my video to the [[FreeCAD_101#Self-Verifying_FreeCAD_Exam_Videos]] article
# [[Maltfield_Log/2020_Q1|2020 Q1 Logs for Michael Altfield]]
# Added my comment to [https://disqus.com/home/discussion/osefreecadinstructionals/freecad_101_open_source_ecology/#comment-3340836890 disqus]
# [[Maltfield_Log/2019_Q4|2019 Q4 Logs for Michael Altfield]]
 
# [[Maltfield_Log/2019_Q3|2019 Q3 Logs for Michael Altfield]]
=Thr June 01, 2017=
# [[Maltfield_Log/2019_Q2|2019 Q2 Logs for Michael Altfield]]
# Updated OSE Wiki with better instructions to install the Assembly 2 Workbench [[FreeCAD Assembly Workbench]]
# [[Maltfield_Log/2019_Q1|2019 Q1 Logs for Michael Altfield]]
# successfully finished sketching polylines w/ constraints to pocket my 2d initials into the 3d xyz cube
# [[Maltfield_Log/2018_Q4|2018 Q4 Logs for Michael Altfield]]
# recorded myself building another xyz cube, initials, pocketed. Sped-up video to 30-seconds, added soundtrack, & [https://www.youtube.com/watch?v=s8jUpl3fgdA uploaded my finished freecad test to youtube]]
# [[Maltfield_Log/2018_Q3|2018 Q3 Logs for Michael Altfield]]
# Created a new repo on my github or my OSE work, and committed/pushed my xyz cube work to [https://github.com/guttersnipe098/ose/tree/master/2017-05.freecadTest here]
# [[Maltfield_Log/2018_Q2|2018 Q2 Logs for Michael Altfield]]
## See [https://github.com/guttersnipe098/ose/blob/master/2017-05.freecadTest/video.fcstd video.fcstd] for the freecad file that was created during the video's recording.
# [[Maltfield_Log/2018_Q1|2018 Q1 Logs for Michael Altfield]]
 
# [[Maltfield_log_2017|2017 Logs for Michael Altfield]]
=Tue May 30, 2017=
# Finished MarthamEngineering's 3-part freecad youtube introduction series
# Began [[https://www.youtube.com/watch?v=RwcFg-anVHE Marcin's 2-part freecad tutorial]]
# Updated OSE Wiki with better instructions to install Fastener's Workbench [[Fasteners Workbench in FreeCAD]]
# Successfully finished building my first xyz cube in FreeCAD
 
=Mon May 29, 2017=
# Began reading freecad documentation, watching videos on, & playing with freecad
# Began MarthamEngineering's FreeCAD Tutorials youtube video series, starting with [[https://www.youtube.com/watch?v=QzfT-ZDp0Wo part 1]]
 
=Sat Apr 22, 2017=
[[Online_conferencing]]

Latest revision as of 03:00, 31 May 2025

Links: OSE_Server. Michael Photo Folder.

Link: https://osedev.org/wiki/Maltfield

Key Wiki Pages

  1. User:Maltfield
  2. OSE Server and OSE Server#Backups
  3. Web server configuration
  4. Wordpress
  5. Mediawiki
  6. Category:CHGs

Logs

In early 2018 I discovered that this page had become the single longest article in our wiki, so I segregated out my work log by year.

In mid-2018, I discovered that my work log had become so large that it exceeded Mediawiki's allowable limits (triggering a "413 Request Entity Too Large" error), so I segregated out my work log by quarter into subpages.


  1. 2025 Q2 Logs for Michael Altfield
  2. 2025 Q1 Logs for Michael Altfield
  3. 2024 Q4 Logs for Michael Altfield
  4. 2024 Q3 Logs for Michael Altfield
  5. 2024 Q1 & Q2 Logs for Michael Altfield
  6. 2023 Logs for Michael Altfield
  7. 2022 Logs for Michael Altfield
  8. 2021 Q3 Logs for Michael Altfield
  9. 2021 Q2 Logs for Michael Altfield
  10. 2020 Q3 Logs for Michael Altfield
  11. 2020 Q2 Logs for Michael Altfield
  12. 2020 Q1 Logs for Michael Altfield
  13. 2019 Q4 Logs for Michael Altfield
  14. 2019 Q3 Logs for Michael Altfield
  15. 2019 Q2 Logs for Michael Altfield
  16. 2019 Q1 Logs for Michael Altfield
  17. 2018 Q4 Logs for Michael Altfield
  18. 2018 Q3 Logs for Michael Altfield
  19. 2018 Q2 Logs for Michael Altfield
  20. 2018 Q1 Logs for Michael Altfield
  21. 2017 Logs for Michael Altfield
Retrieved from "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log&oldid=307970"