Maltfield Log/2019 Q4

From Open Source Ecology
Jump to: navigation, search

My work log from the year 2019 Quarter 4. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

  1. Maltfield_Log
  2. User:Maltfield
  3. Special:Contributions/Maltfield

Tue Dec 31, 2019

  1. I created a backup of our current year's awstats docroots before awstats overwrites them
[root@opensourceecology awstats.opensourceecology.org]# cp -r htdocs htdocs.20191231
[root@opensourceecology awstats.opensourceecology.org]# cd ../awstats.openbuildinginstitute.org/
[root@opensourceecology awstats.openbuildinginstitute.org]# cp -r htdocs htdocs.20191231
[root@opensourceecology awstats.openbuildinginstitute.org]# ls
htdocs  htdocs.20191231
[root@opensourceecology awstats.openbuildinginstitute.org]# du -sh *
3.0M    htdocs
3.0M    htdocs.20191231
[root@opensourceecology awstats.openbuildinginstitute.org]# date
Tue Dec 31 09:46:59 UTC 2019
[root@opensourceecology awstats.openbuildinginstitute.org]# pwd
/var/www/html/awstats.openbuildinginstitute.org
[root@opensourceecology awstats.openbuildinginstitute.org]# 
  1. this is a consequence of the stupid way I setup awstats' cron job
[root@opensourceecology awstats.openbuildinginstitute.org]# cat /etc/cron.d/awstats_generate_static_files 
06 * * * * root /bin/nice /usr/share/awstats/tools/awstats_updateall.pl -configdir=/etc/awstats/ now
16 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/
17 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=seedhome.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/
18 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=fef.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/
19 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/
20 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=wiki.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/
21 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=microfactory.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/
21 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=store.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/
[root@opensourceecology awstats.openbuildinginstitute.org]# 
  1. I added to my TODO list to update awstats' cron job to use a <year> dir in the output dir; that'll fix this issue.
  2. ...
  3. I also sent-off an email to the Auroville Earth Institute asking when would be a good time to meet them in mid-January


Hello,

I will be visiting Auroville for some time in early January, and I would
love to meet with Auroville Earth Institute to learn your techniques for
CEBs and CSEBs.

My name is Michael. I am working with Open Source Ecology. We are
currently designing open source blueprints for civilization. We also
work with Open Building Institute.

 * https://www.opensourceecology.org
 * https://www.openbuildinginstitute.org

One of the machines that we develop is an Earth Brick Press for making CEBs.

 * https://wiki.opensourceecology.org/wiki/CEB_Press

We are also currently working on a prototype modification to our CEB
press that will include a fully automated hammermill soil crusher &
cement/water mixer before loading the mix into our compression chamber.

 * https://wiki.opensourceecology.org/wiki/Soil_Mixer_2019
 * https://wiki.opensourceecology.org/wiki/Soil_Preparation_for_CEB

I would love to meet with you while I'm in Auroville. When would be a
good time to visit in mid-January?


Thank you,

Michael Altfield
Senior System Administrator
PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B

Open Source Ecology
www.opensourceecology.org

Mon Dec 30, 2019

  1. The Discourse team responded to me about their silly huge cookies. Looks like they store client data there. One of their staff suggested that it may be better to move this to redis instead https://meta.discourse.org/t/discourse-session-cookies-400-request-header-or-cookie-too-large/137245
  2. ...
  3. Marcin just sent out a phplist email, but hit a ton of issues with mod_security. Let's whitelist those
  4. first, let's get a list of the mod_security errors specific to our phplist site. Here's the most recent ones
[root@opensourceecology log]# grep 'ModSecurity' httpd/phplist.opensourceecology.org/error_log | tail -n3
[Mon Dec 30 06:41:13.586546 2019] [:error] [pid 2678] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). String match "bytes=0-" at REQUEST_HEADERS:Range. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "428"] [id "958291"] [rev "2"] [msg "Range: field exists and begins with 0."] [data "bytes=0-524287"] [severity "WARNING"] [ver "OWASP_CRS/2.2.9"] [maturity "6"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"] [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "XgmcCfCVv--9NfCMmWfB@QAAAAQ"]
[Mon Dec 30 06:41:41.349145 2019] [:error] [pid 9847] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "phplist.opensourceecology.org"] [uri "/lists/ut.php"] [unique_id "XgmcJdQv-PvExNQwXttQ2AAAAAo"]
[Mon Dec 30 06:49:30.756792 2019] [:error] [pid 9844] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "phplist.opensourceecology.org"] [uri "/lists/ut.php"] [unique_id "Xgmd@jjXTNvlhqNnJNUG7gAAAAY"]
[root@opensourceecology log]# 
  1. Here's a quick command to see all the mod_security rules that triggered by rule id, counted by the number of their occurance and sorted from most common to least common
[root@opensourceecology log]# grep 'ModSecurity' httpd/phplist.opensourceecology.org/error_log | sed 's/^.*\[id "\([^"]*\).*$/\1/' | sort -n | uniq -c | sort -rn
	484 960015
	  9 981242
	  9 958008
	  7 959071
	  7 950109
	  5 960020
	  4 959072
	  1 958291
[root@opensourceecology log]# 
    1. so by far the most common is 484x occurances of rule #960015 = "Request Missing an Accept Header"
    2. the next most common is a tie for 9 occurances of rule#98124 = "Detects classic SQL injection probings 1/2"
    3. And rule #958008 = "Cross-site Scripting (XSS) Attack"
  1. While I was tailing the log, I saw one of those 960015 errors (by far the most common occuring one) pop into the logs. It was for this /lists/ut.php page. I loaded it and, yeah, it's a tracking pixel. That makes sense as to why the clients are missing accept headers; it's probably some email client thing https://phplist.opensourceecology.org/lists/ut.php
[Mon Dec 30 10:52:56.122828 2019] [:error] [pid 28739] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "phplist.opensourceecology.org"] [uri "/lists/ut.php"] [unique_id "XgnXCDikFSEDO3Zy3gVuUAAAAAY"]
  1. I tested one of the images that Marcin said caused a 403, but it added & sent without issues Vid.png
  2. I tested a string he said failed, and it indeed failed for me too
use 1.75 and 3 mm filament interchangeably,
  1. Adding that to the body of the email and pressing "Next" produced a 403 and this to the logs
[Mon Dec 30 10:10:21.953649 2019] [:error] [pid 19509] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i)\\\\b(?i:and)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')\\\\s*?[=]|\\\\b(?i:and)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')\\\\s*?[<>]|\\\\band\\\\b ?(?:\\\\d{1,10}|[\\\\'\\"][^=]{1,10}[\\\\'\\"]) ?[=<>]+|\\\\b(?i:and)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')" at ARGS:message. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "136"] [id "959072"] [rev "2"] [msg "SQL Injection Attack"] [data "Matched Data: and 3 found within ARGS:message: <p><img alt=\\x22\\x22 src=\\x22https://wiki.opensourceecology.org/images/3/31/Vid.png\\x22 /></p>\\x0d\\x0a\\x0d\\x0a<p>use 1.75 and 3 mm filament interchangeably,</p>\\x0d\\x0a\\x0d\\x0a<p> </p>\\x0d\\x0a"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "phplist.opensourceecology.org"] [uri "/lists/admin/"] [unique_id "XgnNDTENrNCvqKpO5DuR9AAAAAQ"]
  1. I whitelisted that rule, here's the diff that OSSEC emailed me immediately
Integrity checksum changed for: '/etc/httpd/conf.d/00-phplist.opensourceecology.org.conf'
Size changed from '2984' to '2991'
What changed:
76c76
< 		SecRuleRemoveById 970901 950001 950120 950901 981173 981317 973300 960024 950911 981231 981248 981245 973338 973304 973306 973333 973344 981257 981240 981246 981243 973336 958057 958006 958049 958051 958056 958011 958030 958039 959073 959151 973301 973302 973308 973314 973331 973315 973330 973327 973322 973348 973321 973335 973334 973332 973347 973316 200004 981172 960915 200003
---
> 		SecRuleRemoveById 970901 950001 950120 950901 981173 981317 973300 960024 950911 981231 981248 981245 973338 973304 973306 973333 973344 981257 981240 981246 981243 973336 958057 958006 958049 958051 958056 958011 958030 958039 959072 959073 959151 973301 973302 973308 973314 973331 973315 973330 973327 973322 973348 973321 973335 973334 973332 973347 973316 200004 981172 960915 200003
Old md5sum was: 'e7de88aeca71933be49f1e640cc45a78'
New md5sum is : 'cb2dab2dbf47f98196bfea264f6b5a32'
Old sha1sum was: 'fc2a186b0751e0c51404856e345b4473da05504b'
New sha1sum is : '59e6bb98c3c40920b2052af4770815216d1a3d9d'
  1. All of the rest of the images he said he had issues with worked for me. I asked him to send me his workflow
  2. In any case, there's only one other ModSecurity rule that was fired in the most recent logs, excluding the one from the pixel (MISSING_HEADER_ACCECPT)
[root@opensourceecology log]# grep 'ModSecurity' httpd/phplist.opensourceecology.org/error_log | sed 's/^.*\[id "\([^"]*\).*$/\1/' | grep -vi 960015
960020
959072
959072
[root@opensourceecology log]# 
  1. that id = 960020 appears to just be a NOTICE that the request included the "Pragma" header but not the "Cache-Control" header. I don't think that would be an issue.
--4dcba435-A--                                                                                                                                                                            
[30/Dec/2019:10:09:30 +0000] XgnM2h1TkKX56GufLCph1QAAAAA 127.0.0.1 56908 127.0.0.1 8000                                                                                                   
--4dcba435-B--                                                                                                                                                                            
GET /category/steam-engine-construction-set/ HTTP/1.1                                                                                                                                     
X-Real-IP: 157.55.39.193                                                                                                                                                                  
X-Forwarded-Proto: https                                                                                                                                                                  
X-Forwarded-Port: 443                                                                                                                                                                     
Host: www.opensourceecology.org                                                                                                                                                           
Pragma: no-cache                                                                                                                                                                          
Accept: */*                                                                                                                                                                               
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)                                                                                                       
X-Forwarded-For: 157.55.39.193, 127.0.0.1, 127.0.0.1                                                                                                                                      
Accept-Encoding: gzip                                                                                                                                                                     hash: #www.opensourceecology.org                                                                                                                                                          
X-Varnish: 77278507                                                                                                                                                                       
                                                                                                                                                                                          
--4dcba435-F--                                                                                                                                                                            
HTTP/1.1 403 Forbidden                                                                                                                                                                    
Content-Length: 241                                                                                                                                                                       
Content-Type: text/html; charset=iso-8859-1                                                                                                                                               
                                                                                                                                                                                          
--4dcba435-E--                                                                                                                                                                            
                                                                                                                                                                                          
--4dcba435-H--
Message: Access denied with code 403 (phase 2). String match "HTTP/1.1" at REQUEST_PROTOCOL. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "399"] [id "960020"] [rev "2"] [msg "Pragma Header requires Cache-Control Header for HTTP/1.1 requests."] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "6"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"]
Action: Intercepted (phase 2)                                                                                                                                                             
Stopwatch: 1577700570285846 8089 (- - -)                                                                                                                                                  
Stopwatch2: 1577700570285846 8089; combined=320, p1=282, p2=24, p3=0, p4=0, p5=14, sr=62, sw=0, l=0, gc=0                                                                                 
Response-Body-Transformed: Dechunked                                                                                                                                                      
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.                                                                                                    
Server: Apache                                                                                                                                                                            
Engine-Mode: "ENABLED"
--4dcba435-Z--              
  1. oh, well, that's from www.opensourceecology.org. Here's the most recent one from phplist.opensourceecology.org
--7e84b452-A--                                                                                                                                                                            
[30/Dec/2019:09:17:01 +0000] XgnAjVKRjngP0Kz-535QIwAAAAs 127.0.0.1 44750 127.0.0.1 8000                                                                                                   
--7e84b452-B--                                                                                                                                                                            
GET /lists/ut.php?m=46&u=9e7c1f1611704d32a5ae958fb67c21fb HTTP/1.1                                                                                                                        
X-Real-IP: 46.135.15.135                                                                                                                                                                  
X-Forwarded-Proto: https                                                                                                                                                                  
X-Forwarded-Port: 443                                                                                                                                                                     
Host: phplist.opensourceecology.org                                                                                                                                                       
Pragma: no-cache                                                                                                                                                                          
User-Agent: Mozilla/5.0 (Linux; Android 8.0.0; SM-N950F Build/R16NW; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 Mobile Safari/537.36                     
Accept: image/webp,image/apng,image/*,*/*;q=0.8                                                                                                                                           
Accept-Language: en-GB,en-US;q=0.9                                                                                                                                                        
X-Requested-With: me.bluemail.mail                                                                                                                                                        
X-Forwarded-For: 46.135.15.135, 127.0.0.1, 127.0.0.1                                                                                                                                      
Accept-Encoding: gzip                                                                                                                                                                     
hash: #phplist.opensourceecology.org                                                                                                                                                      
X-Varnish: 76200967                                                                                                                                                                       
                                                                                                                                                                                          
--7e84b452-F--                                                                                                                                                                            
HTTP/1.1 403 Forbidden                                                                                                                                                                    
Content-Length: 214                                                                                                                                                                       
Content-Type: text/html; charset=iso-8859-1                                                                                                                                               
                                                                                                                                                                                          
--7e84b452-E--                                                                                                                                                                            
                                                                                                                                                                                          
--7e84b452-H--                                                                                                                                                                            
Message: Access denied with code 403 (phase 2). String match "HTTP/1.1" at REQUEST_PROTOCOL. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "399"] [id "960020"] [rev "2"] [msg "Pragma Header requires Cache-Control Header for HTTP/1.1 requests."] [severity "NOTICE"] [ver "OWASP_CRS/2.2.9"] [maturity "6"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"]                                                                                                                                        
Action: Intercepted (phase 2)                                                                                                                                                             
Apache-Handler: php5-script                                                                                                                                                               
Stopwatch: 1577697421335612 8009 (- - -)                                                                                                                                                  Stopwatch2: 1577697421335612 8009; combined=316, p1=275, p2=22, p3=0, p4=0, p5=19, sr=62, sw=0, l=0, gc=0                                                                                 Response-Body-Transformed: Dechunked                                                                                                                                                      
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.                                                                                                    
Server: Apache                                                                                                                                                                            
Engine-Mode: "ENABLED"                                                                                                                                                                    
                                                                                                                                                                                          
--7e84b452-Z--              
  1. whatever, that's someone on safari I guess violating soeme standard viewing our email. Anyway, shouldn't be an issue.
  2. Oh, but there's a log rotated for today already; that one has more
[root@opensourceecology log]# cat httpd/phplist.opensourceecology.org/error_log-20191230 | grep 'ModSecurity' | sed 's/^.*\[id "\([^"]*\).*$/\1/' | grep -vi 960015 | grep -vi 960020 | uniq -c | sort -rn
	  9 981242
	  9 958008
	  7 959071
	  5 950109
	  2 959072
	  2 959072
	  2 958008
	  2 950109
	  1 958291
[root@opensourceecology log]# x
  1. ...
  2. I just got an email from Marcin saying he's resizing the images to 500 on the X; I tried doing that too, and it worked. But I checked the logs again in-case he recently put something there
[root@opensourceecology log]# cat httpd/phplist.opensourceecology.org/error_log | grep 'ModSecurity' | sed 's/^.*\[id "\([^"]*\).*$/\1/' | uniq
960015
960020
960015
959072
960015
[root@opensourceecology log]# 
  1. None of those work. I did some more poking around, and I got it to trigger a 403 when I clicked the "Image Button" button (in the second row) instead of the "Image" button in the (second-to-last row). Please don't ask me what the difference is *shrug*. The logs say this
[Mon Dec 30 11:23:06.013512 2019] [:error] [pid 27948] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "<input\\\\b.*?\\\\btype\\\\b\\\\W*?\\\\bimage\\\\b" at ARGS:message. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "229"] [id "958008"] [rev "2"] [msg "Cross-site Scripting (XSS) Attack"] [data "Matched Data: <input alt=\\x22\\x22 src=\\x22https://wiki.opensourceecology.org/images/b/bf/soimixerhyd.jpg\\x22 style=\\x22width: 1244px; height: 820px;\\x22 type=\\x22image found within ARGS:message: <p><input alt=\\x22\\x22 src=\\x22https://wiki.opensourceecology.org/images/b/bf/soimixerhyd.jpg\\x22 style=\\x22width: 1244px; height: 820px;\\x22 type=\\x22image\\x22 /><iframe frameborder=\\x220\\x22 scrolling=\\x22no\\x22 src=\\x22https://wiki.opensourceecology.org/images/b/bf/soimixerhyd.jpg\\x22 width=\\x22500\\x22></ifram..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "phplist.opensourceecology.org"] [uri "/lists/admin/"] [unique_id "XgneGWMB55Go@mg1L473bwAAAAA"]
  1. So I whitelisted "958008" too
  2. Marcin said it now works for him over email.
  3. ...
  4. I spent some time getting screenshots for the 2019 year from awstats and putting them on the wiki like we have for Munin.
    1. https://wiki.opensourceecology.org/wiki/Awstats
    2. https://wiki.opensourceecology.org/wiki/Category:Awstats_Graphs
  5. I also drafted a summary of the site's stats in 2019 and sent it to Marcin & Catarina
Hey All,

Before the year closes, I gathered some stats about our sites and stored
them to the wiki.

 * https://wiki.opensourceecology.org/wiki/Category:Awstats_Graphs

A snapshot of our most popular sites:

wiki.opensourceecology.org received 10,695,729 hits YTD in 2019 from
452,260 unique visitors. The most popular month was in August, where we
got 1,164,161 hits from 48,918 unique visitors. The least popular month
was February, where we got 828,342 hits from 30,326 unique visitors.

Here's our top pages on the wiki:

/wiki/Main_Page
/wiki/Cost_of_Living
/wiki/OSE_Machine_Design_Guide
/wiki/Special:RequestAccount
/wiki/Global_Village_Construction_Set
/wiki/Civilization_Starter_Kit_DVD_v0.01
/wiki/Aquaponics

 *
https://awstats.opensourceecology.org:4443/awstats.wiki.opensourceecology.org.html

www.opensourceecology.org received 9,413,080 hits YTD in 2019 from
360,643 unique visitors. The most popular month was also August, and the
least popular December (so far).

Here's the top pages on osemain:

/gvcs/
/gvcs/gvcs-machine-index/
/about-videos-3/
/ceb-microhouse-build-in-belize/
/marcin-jakubowski/
/web-developers-for-better-true-fans-campaign/
/portfolio/microhouse/

 *
https://awstats.opensourceecology.org:4443/awstats.www.opensourceecology.org.html

www.openbuildinginstitute.org received 3,949,352 hits YTD in 2019 from
84,220 unique visitors. The most popular month was in May, where we got
357,333 hits from 8,668 unique visitors. The least popular month is
currently December, where we've so-far received 238,738 from 3,795
unique visitors.

Here's our top pages for OBI:

/use/
/buildings/
/how-it-works/
/about-what-we-do/
/structures/
/library-modules/
/portfolio/studio-12x16/

 *
https://awstats.openbuildinginstitute.org:4443/awstats.www.openbuildinginstitute.org.html

Think we can double these numbers in 2020? Happy new years :)

Cheers,

Michael Altfield
Senior System Administrator
PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B

Open Source Ecology
www.opensourceecology.org
  1. I wanted to get uptime stats from statuscake, but it looks like all our tests were paused. Strange, because we're still getting 7-day stats on our status page though *shrug* http://status.opensourceecology.org/

Sun Dec 29, 2019

  1. On my last call with Marcin, he again asked what bottlenecks we would hit on our server if we suddenly had thousands of OSE developers contributing in September 2020. I told him again that disk was my biggest concern
  2. Considering adding docker to our server, I bean to wonder how that would increase our disk usage. Indeed, a common issue with implementing docker in prod is disk fill because of all the old docker images that pile up without implementing some sort of cleanup cron job. Adding such a cron job is on my TODO list, but it's still going to add a reasonably large tax on our disks just by utilizing Discourese & docker.
  3. I checked our disks. Currently prod is using 77/197 usable space = 41%. I think 70% usage is when we'd want to begin migrating to a server with bigger disks. That migration would be a huge effort and be monetarily costly.
[maltfield@opensourceecology ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        197G   77G  111G  41% /
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G  8.0K   32G   1% /dev/shm
tmpfs            32G  3.2G   29G  11% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/md1        488M  289M  174M  63% /boot
tmpfs           6.3G     0  6.3G   0% /run/user/0
tmpfs           6.3G     0  6.3G   0% /run/user/1005
[maltfield@opensourceecology ~]$ 
  1. In any case, I decided to message hetzner to see what our options were
  2. first of all, when logging-into our hetzner admin page, I saw they have an offer for a "Dedicated Root Server AX41-NVMe" server at the same price as what we're currently paying (39 EUR/mo) with twice the size of RAID1'd disks (2x 512G NVMe SSD vs 2x 250G SSD). RAM would be the same. Processor would probably be faster. So merely "upgrading" our monolithic server at no additional montly cost (plus 39 EUR one-time setup fee) is one option.
  3. I noticed that we have a tab on our wui admin page with hetzner called "storage box". Motherfucker, it says "inclusive". Looks like it comes with our Dedicated Server plan & is SSD with 100G storage.
  4. wow, they got back to me pretty fast. Looks like our server can have two additional drives from 0.5T - 12T added. They have a ton of options from NVMe SSDs to fat & cheap spinning disks. The easy fix would be to just add two 12T drives RAID 1'd for 26.5 EUR. The better solution would be to migrate to another server of better specs and the same price, perhaps setting that one up with a software RAID5 with all 4x disks included. https://wiki.hetzner.de/index.php/Root_Server_Hardware/en#Drives
  5. It looks like the 100G storage box isn't going to be decent enough for NFS mount; I think it's meant for backups https://wiki.hetzner.de/index.php/Storage_Boxes/en
  6. But even then, we can move our encrypted backups off to the Storage box, and that would save us 18G of space
[root@opensourceecology ~]# du -sh /home/b2user/sync*
18G	/home/b2user/sync
18G	/home/b2user/sync.old
[root@opensourceecology ~]# 
  1. I also updated our wiki OSE_Server page with info on this disk bottleneck dedicated server solutions https://wiki.opensourceecology.org/wiki/OSE_Server#Dedicated_Servers

Sat Dec 28, 2019

  1. let's attempt to update discourse
  2. first let's get the current versions
    1. It says our installed Discourse version is "2.4.0.beta8 (8106d94c05)" https://discourse.opensourceecology.org/admin
    2. The upgrade page itself listed the version of discourse_manager (I guess that's a plugin) as "2c89085" https://discourse.opensourceecology.org/admin/upgrade
  3. Now let's attempt an upgrade
    1. first we make a backup
[root@osestaging1 ~]# export vhostDir="/var/discourse/"
[root@osestaging1 ~]# 
[root@osestaging1 ~]# # verify
[root@osestaging1 ~]# echo "${vhostDir}"
/var/discourse/
[root@osestaging1 ~]# ls -lah "${vhostDir}"
total 1.1M
drwxr-xr-x. 12 root root 4.0K Dec 17 11:36 .
drwxr-xr-x. 23 root root 4.0K Oct 28 12:07 ..
drwxr-xr-x.  2 root root 4.0K Oct 28 12:07 bin
drwxr-xr-x.  2 root root 4.0K Dec 17 11:29 cids
drwxr-xr-x.  2 root root 4.0K Dec 17 12:18 containers
-rwxr-xr-x.  1 root root  12K Oct 28 12:07 discourse-doctor
-rwxr-xr-x.  1 root root  21K Dec 16 12:20 discourse-setup
-rw-r--r--.  1 root root 2.4K Nov  7 08:59 docker-ce.repo
-rw-r--r--.  1 root root 1.6K Nov  7 09:05 docker.gpg
-rw-r--r--.  1 root root  13K Oct  7 23:35 get-docker.sh
drwxr-xr-x.  8 root root 4.0K Dec 17 07:38 .git
-rw-r--r--.  1 root root  309 Oct 28 12:07 .gitignore
drwxr-xr-x.  8 root root 4.0K Nov 18 08:17 image
-rw-r--r--.  1 root root  13K Oct  7 23:35 index.html
-rw-r--r--.  1 root root  76K Oct 28 13:55 install.sh
-rwxr-xr-x.  1 root root  23K Dec 17 11:36 launcher
-rwxr-xr-x.  1 root root  23K Nov 18 12:13 launcher.20191118_122249
-rwxr-xr-x.  1 root root  23K Nov 18 12:03 launcher.20191118.orig
-rwxr-xr-x.  1 root root  30K Nov 26 16:02 launcher.20191217
-rwxr-xr-x.  1 root root  23K Dec 17 07:38 launcher.20191217_074503
-rwxr-xr-x.  1 root root  23K Dec 17 07:45 launcher.20191217_104906
-rwxr-xr-x.  1 root root  23K Nov 18 12:02 launcher.new
-rwxr-xr-x.  1 root root  24K Nov 26 10:25 launcher.old
drwxr-xr-x.  5 root root 4.0K Nov 12 11:11 libbrotli
-rw-r--r--.  1 root root 1.1K Oct 28 12:07 LICENSE
-rw-r--r--.  1 root root 664K Nov 18 11:43 output.log
-rw-r--r--.  1 root root 8.7K Oct 28 12:07 README.md
drwxr-xr-x.  2 root root 4.0K Dec 17 11:18 samples
drwxr-xr-x.  2 root root 4.0K Oct 28 12:07 scripts
drwxr-xr-x.  3 root root 4.0K Nov  7 11:27 shared
drwxr-xr-x.  3 root root 4.0K Dec 17 10:48 templates
-rw-r--r--.  1 root root 1.3K Oct 28 12:07 Vagrantfile
[root@osestaging1 ~]# 
stamp=`date +%Y%m%d_%T`
tmpDir="/var/tmp/discourseUpgrade.${stamp}"
mkdir "${tmpDir}"
chown root:root "${tmpDir}"
chmod 0700 "${tmpDir}"
pushd "${tmpDir}"

# discourse backup (db & uploaded files only)
nice rm -rf /var/discourse/shared/standalone/backups/default/*.tar.gz
time nice docker exec discourse_ose discourse backup
nice mv /var/discourse/shared/standalone/backups/default/*.tar.gz ${tmpDir}/

# files backup (all discourse files)
time nice tar --exclude "${vhostDir}/shared/standalone/postgres_data" --exclude "${vhostDir}/shared/standalone/postgres_data/uploads" --exclude "${vhostDir}/shared/standalone/backups" -czf ${tmpDir}/discourse_files.${stamp}.tar.gz /var/discourse/*
[root@osestaging1 discourseUpgrade.20191228_14:00:30]# date
Sat Dec 28 14:09:31 UTC 2019
[root@osestaging1 discourseUpgrade.20191228_14:00:30]# pwd
/var/tmp/discourseUpgrade.20191228_14:00:30
[root@osestaging1 discourseUpgrade.20191228_14:00:30]# ls
discourse-2019-12-28-140114-v20191211170000.tar.gz  discourse_files.20191228_14:00:30.tar.gz
[root@osestaging1 discourseUpgrade.20191228_14:00:30]# du -sh *
52M     discourse-2019-12-28-140114-v20191211170000.tar.gz
78M     discourse_files.20191228_14:00:30.tar.gz
[root@osestaging1 discourseUpgrade.20191228_14:00:30]# 
  1. now we `git pull` the changes to /var/discourse on the docker host (osestaging1). Ok, that failed because of our changes to "launcher"
[root@osestaging1 ~]# pushd "${vhostDir}"
/var/discourse ~
[root@osestaging1 discourse]# git pull
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 1), reused 2 (delta 1), pack-reused 0
Unpacking objects: 100% (3/3), done.
From https://github.com/discourse/discourse_docker
   026a664..1b3dd3a  master     -> origin/master
Updating 026a664..1b3dd3a
error: Your local changes to the following files would be overwritten by merge:
		launcher
Please, commit your changes or stash them before you can merge.
Aborting
[root@osestaging1 discourse]# 
  1. here's a list of all the changes I've made; namely the modified three at the top should probably be reverted before the git pull then updated after the git pull
[root@osestaging1 discourse]# git status
# On branch master
# Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
#   (use "git pull" to update your local branch)
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   image/base/Dockerfile
#       modified:   image/base/install-nginx
#       modified:   launcher
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       docker-ce.repo
#       docker.gpg
#       get-docker.sh
#       image/base/install-nginx.20191112
#       image/base/install-nginx.20191112.orig
#       image/base/install-nginx.20191125_122229.orig
#       image/base/install-nginx.20191125_123516.orig
#       image/base/runit-1.d-01-iptables
#       index.html
#       install.sh
#       launcher.20191118.orig
#       launcher.20191118_122249
#       launcher.20191217
#       launcher.20191217_074503
#       launcher.20191217_104906
#       launcher.new
#       launcher.old
#       libbrotli/
#       output.log
#       templates/iptables.template.yml
#       templates/web.modsecurity.template.yml
no changes added to commit (use "git add" and/or "git commit -a")
[root@osestaging1 discourse]# 
  1. the changes to Dockerfile were not needed; I'm updating the documentation to move the 'launcher' and 'install-nginx' scripts out of the way
[root@osestaging1 discourse]# mv "${vhostDir}/launcher" "${vhostDir}/launcher.`date "+%Y%m%d_%H%M%S"`"
[root@osestaging1 discourse]# mv "${vhostDir}/image/base/install-nginx" "${vhostDir}/image/base/install-nginx.`date "+%Y%m%d_%H%M%S"`"
[root@osestaging1 discourse]# 
[root@osestaging1 discourse]# pwd
/var/discourse
[root@osestaging1 discourse]# git pull
Updating 026a664..1b3dd3a
Fast-forward
 launcher | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
[root@osestaging1 discourse]# 
  1. then I re-updated the install-nginx script to add mod_security support
[root@osestaging1 discourse]# pushd "${vhostDir}/image/base"
/var/discourse/image/base /var/discourse ~
[root@osestaging1 base]# cp install-nginx install-nginx.`date "+%Y%m%d_%H%M%S"`.orig
[root@osestaging1 base]# 
[root@osestaging1 base]# # add a block to checkout the the modsecurity nginx module just before downloading the nginx source
[root@osestaging1 base]# grep 'ModSecurity' install-nginx || sed -i 's%\(curl.*nginx\.org/download.*\)%# mod_security --maltfield\napt-get install -y libmodsecurity-dev modsecurity-crs\ncd /tmp\ngit clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git\n\n\1%' install-nginx
[root@osestaging1 base]# 
[root@osestaging1 base]# # update the configure line to include the ModSecurity module checked-out above
[root@osestaging1 base]# sed -i '/ModSecurity/! s%^[^#]*./configure \(.*nginx.*\)%#./configure \1\n./configure \1 --add-module=/tmp/ModSecurity-nginx%' install-nginx
[root@osestaging1 base]# 
[root@osestaging1 base]# # add a line to cleanup section
[root@osestaging1 base]# grep 'rm -fr /tmp/ModSecurity-nginx' install-nginx || sed -i 's%\(rm -fr.*/tmp/nginx.*\)%rm -fr /tmp/ModSecurity-nginx\n\1%' install-nginx
[root@osestaging1 base]# 
[root@osestaging1 base]# popd
/var/discourse ~
[root@osestaging1 discourse]# 
  1. And the necessary change to 'launcher'
[root@osestaging1 discourse]# pushd "${vhostDir}"
/var/discourse /var/discourse ~
[root@osestaging1 discourse]# 
[root@osestaging1 discourse]# # replace the line "image="discourse/base:<version>" with 'image="discourse_ose"'
[root@osestaging1 discourse]# grep 'discourse_ose' launcher || sed --in-place=.`date "+%Y%m%d_%H%M%S"` '/base_image/! s%^\(\s*\)image=\(.*\)$%#\1image=\2\n\1image="discourse_ose"%' launcher
[root@osestaging1 discourse]# 
[root@osestaging1 discourse]# popd
/var/discourse ~
[root@osestaging1 discourse]# 
  1. git diff confirms the changes
[root@osestaging1 discourse]# git diff
diff --git a/image/base/install-nginx b/image/base/install-nginx
index 7b91333..172d795 100755
--- a/image/base/install-nginx
+++ b/image/base/install-nginx
@@ -18,6 +18,11 @@ cd /tmp
 # this is the reason we are compiling by hand...
 git clone https://github.com/google/ngx_brotli.git
 
+# mod_security --maltfield
+apt-get install -y libmodsecurity-dev modsecurity-crs
+cd /tmp
+git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git
+
 curl -O https://nginx.org/download/nginx-$VERSION.tar.gz
 tar zxf nginx-$VERSION.tar.gz
 cd nginx-$VERSION
@@ -31,13 +36,15 @@ apt-mark hold nginx
 cd /tmp/ngx_brotli && git submodule update --init && cd /tmp/nginx-$VERSION
 
 # ignoring depracations with -Wno-deprecated-declarations while we wait for this https://github.com/google/ngx_brotli/issues/39#issuecomment-254093378
-./configure --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --with-ld-opt='-Wl,-Bsymbolic-
+#./configure --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --with-ld-opt='-Wl,-Bsymbolic
+./configure --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations' --with-ld-opt='-Wl,-Bsymbolic-
 
 make install
 
 mv /usr/share/nginx/sbin/nginx /usr/sbin
 
 cd /
+rm -fr /tmp/ModSecurity-nginx
 rm -fr /tmp/nginx
 rm -fr /tmp/libbrotli
 rm -fr /tmp/ngx_brotli
diff --git a/launcher b/launcher
index 41e7c72..0b8b5c8 100755
--- a/launcher
+++ b/launcher
@@ -88,7 +88,8 @@ git_rec_version='1.8.0'
 config_file=containers/"$config".yml
 cidbootstrap=cids/"$config"_bootstrap.cid
 local_discourse=local_discourse
-image="discourse/base:2.0.20191219-2109"
+#image="discourse/base:2.0.20191219-2109"
+image="discourse_ose"
 docker_path=`which docker.io 2> /dev/null || which docker`
 git_path=`which git`
  1. now we can rebuild the Discourse Docker image with nginx mod_security
[root@osestaging1 discourse]# pushd "${vhostDir}/image/base"
/var/discourse/image/base /var/discourse ~
[root@osestaging1 base]# 
[root@osestaging1 base]# # force a fresh build (no-cache) so the `git pull` lines will trigger
[root@osestaging1 base]# # note this will take a *ridiculously* long time; the Discourse team compiles many packages from source :(
[root@osestaging1 base]# time nice docker build --no-cache --network=host --tag 'discourse_ose' /var/discourse/image/base/
...
Removing intermediate container 9a74706741d2
 ---> f360219e7107
Successfully built f360219e7107
Successfully tagged discourse_ose:latest

real    40m46.372s
user    0m0.998s
sys     0m0.659s
[root@osestaging1 base]# 
[root@osestaging1 base]# popd
/var/discourse ~
[root@osestaging1 discourse]# 
  1. And finally we rebuild the Discourse app. Ugh, that failed with that old stupid message
[root@osestaging1 discourse]# ${vhostDir}/launcher rebuild discourse_ose
...
+ /bin/docker rm discourse_ose
Error response from daemon: container 038ea7a12fa5882a16a22da89ccd5d8b04cda241ea80cf0b017c76b1d34a76ee: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/799e97d530bd7cb1d8d93aeb685f13daf0e4bfbe272cab76aa0f017c1a04e7b3/merged: device or resource busy
debug2: channel 0: window 999139 sent adjust 49437

starting up existing container
+ /bin/docker start discourse_ose
Error response from daemon: container is marked for removal and cannot be started
Error: failed to start containers: discourse_ose
[root@osestaging1 discourse]# 
  1. I guess it's unhappy about this one that says "Removal In Progress"
[root@osestaging1 discourse]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES
038ea7a12fa5        902ab1153546        "/sbin/boot"             11 days ago         Removal In Progress                            discourse_ose
5cc0db30940b        940c0024cbd7        "/bin/bash -c 'cd /p…"   11 days ago         Exited (1) 11 days ago                         thirsty_borg
24a1f9f4c038        6a959e2d597c        "/bin/bash"              4 weeks ago         Exited (1) 4 weeks ago                         peaceful_leavitt
6932865cc6a1        6a959e2d597c        "/bin/bash"              4 weeks ago         Exited (1) 4 weeks ago                         friendly_grothendieck
fce75ef5ce06        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (0) 4 weeks ago                         gifted_booth
03ea184c205e        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (127) 4 weeks ago                       clever_solomon
6bd5bb0ab7b5        940c0024cbd7        "whoami"                 4 weeks ago         Exited (0) 4 weeks ago                         upbeat_booth
4fbcfcc1e05f        940c0024cbd7        "echo hello"             4 weeks ago         Created                                        sweet_lalande
88d916eb12b0        940c0024cbd7        "echo hello"             4 weeks ago         Created                                        goofy_allen
4a3b6e123460        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (1) 4 weeks ago                         adoring_mirzakhani
ef4f90be07e6        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (0) 4 weeks ago                         awesome_mcclintock
580c0e430c47        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (130) 4 weeks ago                       naughty_greider
4bce62d2e873        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Created                                        boring_lehmann
6d4ef0ebb57d        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Created                                        loving_davinci
4d5c8b2a90e0        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         quizzical_mestorf
34a3f6146a1d        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         epic_williamson
f0a73d8db0db        940c0024cbd7        "iptables -L"            4 weeks ago         Created                                        dazzling_beaver
4f34a5f5ee65        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         quizzical_haslett
0980ad174804        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         wonderful_tereshkova
79413047322f        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Created                                        naughty_proskuriakova
ba00edad459a        940c0024cbd7        "sudo apt-get instal…"   4 weeks ago         Created                                        quizzical_burnell
7364dbb52542        940c0024cbd7        "sudo apt-get instal…"   4 weeks ago         Created                                        cocky_bhaskara
9d0e485beba0        940c0024cbd7        "sudo apt-get instal…"   4 weeks ago         Created                                        nervous_greider
75394a9e553f        940c0024cbd7        "/usr/sbin/iptables …"   4 weeks ago         Created                                        admiring_cori
8c59607a7b23        940c0024cbd7        "iptables -L"            4 weeks ago         Created                                        silly_buck
92a929061a43        940c0024cbd7        "bash"                   4 weeks ago         Exited (0) 4 weeks ago                         sleepy_cohen
0d4c01df1acb        940c0024cbd7        "bash"                   4 weeks ago         Exited (0) 4 weeks ago                         busy_satoshi
3557078bec62        940c0024cbd7        "/bin/bash -c 'echo …"   4 weeks ago         Exited (0) 4 weeks ago                         busy_sammet
56360e585353        bd5b8ac7ac36        "/bin/sh -c 'apt upd…"   4 weeks ago         Exited (100) 4 weeks ago                       youthful_hermann
53bbee438a5e        9b33df0cef8e        "/bin/sh -c 'apt upd…"   5 weeks ago         Exited (127) 5 weeks ago                       awesome_newton
[root@osestaging1 discourse]# 
  1. well, for some reason I was just able to remove it myself without issue..
[root@osestaging1 discourse]# docker rm 038ea7a12fa5
038ea7a12fa5
[root@osestaging1 discourse]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES
5cc0db30940b        940c0024cbd7        "/bin/bash -c 'cd /p…"   11 days ago         Exited (1) 11 days ago                         thirsty_borg
24a1f9f4c038        6a959e2d597c        "/bin/bash"              4 weeks ago         Exited (1) 4 weeks ago                         peaceful_leavitt
6932865cc6a1        6a959e2d597c        "/bin/bash"              4 weeks ago         Exited (1) 4 weeks ago                         friendly_grothendieck
fce75ef5ce06        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (0) 4 weeks ago                         gifted_booth
03ea184c205e        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (127) 4 weeks ago                       clever_solomon
6bd5bb0ab7b5        940c0024cbd7        "whoami"                 4 weeks ago         Exited (0) 4 weeks ago                         upbeat_booth
4fbcfcc1e05f        940c0024cbd7        "echo hello"             4 weeks ago         Created                                        sweet_lalande
88d916eb12b0        940c0024cbd7        "echo hello"             4 weeks ago         Created                                        goofy_allen
4a3b6e123460        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (1) 4 weeks ago                         adoring_mirzakhani
ef4f90be07e6        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (0) 4 weeks ago                         awesome_mcclintock
580c0e430c47        940c0024cbd7        "/bin/bash"              4 weeks ago         Exited (130) 4 weeks ago                       naughty_greider
4bce62d2e873        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Created                                        boring_lehmann
6d4ef0ebb57d        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Created                                        loving_davinci
4d5c8b2a90e0        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         quizzical_mestorf
34a3f6146a1d        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         epic_williamson
f0a73d8db0db        940c0024cbd7        "iptables -L"            4 weeks ago         Created                                        dazzling_beaver
4f34a5f5ee65        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         quizzical_haslett
0980ad174804        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Exited (0) 4 weeks ago                         wonderful_tereshkova
79413047322f        940c0024cbd7        "/usr/bin/apt-get in…"   4 weeks ago         Created                                        naughty_proskuriakova
ba00edad459a        940c0024cbd7        "sudo apt-get instal…"   4 weeks ago         Created                                        quizzical_burnell
7364dbb52542        940c0024cbd7        "sudo apt-get instal…"   4 weeks ago         Created                                        cocky_bhaskara
9d0e485beba0        940c0024cbd7        "sudo apt-get insta