View source for Maltfield Log/2019 Q1

My work log from the year 2019 Quarter 1. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

__TOC__

=See Also=
# [[Maltfield_Log]]
# [[User:Maltfield]]
# [[Special:Contributions/Maltfield]]

=Sat Feb 02, 2019=
# Marcin responded to my email about the updated wordpress post about phplist
## he asked if I could make the link to the privacy policy open it in a new window, which I totally agree with. I did this by changing the attribute #4 = "I agree to the OSE Privacy Policy." anchor link to include the attribute "target" with value "_blank" in the phplist wui. I also changed it in the html within the wp post
<pre>
I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy' target='_blank'>Privacy Policy</a>.
</pre>
## Marcin asked me to change the response after signup to include a period in the sentence "Thanks, you have been added to our newsletter."
### he also noted that the javascript includes a variable named "successMessage" with a distinct message. Indeed, the text that's displayed comes as a response to the ajax query to the phplist server, and this response is configurable from the phplist wui at Config -> Settings -> "Text to display when subscription with an AJAX request was successful". I changed it to:
<pre>
<h3>Thanks, you have been added to our newsletter.</h3><p>You will receive an email to confirm your subscription. Please click the link in the email to confirm.</p>
</pre>
### that actually didn't work. it looks like that's a default setting, but there's also per-subscribe-list settings for this response. I changed this via Config -> Subscribe Pages -> Edit (for id=3, which matches the URL used in the html of the post) -> "Text to display when subscription with an AJAX request was successful". I changed this to:
<pre>
<h3>Thanks, you have been added to our newsletter.</h3><p>You will receive an email to confirm your subscription. Please click the link in the email to confirm.</p>
</pre>
## Marcin provided a description of the Design Sprints newsletter as well, which I added to the subscribe list's config in phplist
# I updated the wiki documentation's example of an ajax form with the changes we made https://wiki.opensourceecology.org/wiki/Phplist#Example_Form_for_OSEmail

# ...

# The phplist team asked me to cherry pick a git commit for a PR I submitted, but I couldn't get it to work; I followed-up https://github.com/phpList/phplist3/pull/445#issuecomment-460923780


=Sat Jan 26, 2019=
# I logged into backblaze b2 to ensure that my nightly backups from hetnzer2 are still coming in after I've deprecated the dreamhost backup script & replaced it with the backblaze one
# I see that there's a file named "daily_hetzner2_20190126_072001.tar.gpg (started large file)"  which I guess is what it looks like when a backup is _currently_ being uploaded.
# I also see that there's a file from the 25th, yesterday--and the days leading up to it. My change was last week, so this looks good.
# The whole 'ose-server-backups' bucket is 190.5 GB with 16 files
# The billing section shows that we paid our $0.78 fee for the first month, and that it's estimating a $0.28 fee for the second month. pretty fucking good.
# I also confirmed that the backup log file (/var/log/backups/backup.log) contains my new comments, which makes the `time` outputs much more useful. And the --noProgress argument has worked wonderfully--the log file is now showing a size of 4kb in du.

# ...

# Marcin drafted a new post to our main ose wordpress site about or transition to phplist. We want to publish this before issuing the repermission campaign.
# I edited the post to add the ajax form in the post, so that new users just learning about the OSEmail newsletter could signup with phplist
## I hit some modsecurity false-positives
### 958052 xss
### 958407 xss
### 973305 xss
### 973307 xss
### 973307 xss
## even though I'm editing the post in the "text" tab (not "visual"), wordpress is still adding invisible "<p>" tags into thet post. For example, it does it in the middle of a javascript function, which creates syntax errors and compeletly breaks the ajax form
### I went to my user profile & checked the box "DIsable the visual editorwhen editing"
### I removed my ajax form block. saved. added it again. saved. It's still injecting these paragraph tags!
<pre>
...
</noscript></p>
<p><script type="text/javascript">
function checkform() {</p>
<p>	// first, clear the response div from the previous attempts results
	jQuery("#result").empty();</p>
...
</pre>
### I found some documentation on this behaviour here https://codex.wordpress.org/Function_Reference/wpautop#Disabling_the_filter
### We probably don't want to wholesale disable it for the entire blog, so the best option is probably to use this simple plugin https://wordpress.org/plugins/wpautop-control/
### I installed & activated this plugin with wp-cli for the osemain 
<pre>
[maltfield@hetzner2 htdocs]$ sudo -u wp -i wp --path=/var/www/html/www.opensourceecology.org/htdocs plugin install --activate wpautop-control
...
</pre>
### # To enable it, I had to go to edit the post and enable the "Custom Fields" checkbox under "Screen Options" at the top. Then I scrolled down to the "Custom Fields" -> "Add New Custom Field:" section, clicked "Enter new", and typed "wpautop" for the Name and "false" for the Value.
### When I reloaded the post's preview, the paragraphs were all running together. Success! I manually added the necessary <p> tags around Marcin's paragraphs. I saved, refreshed the preview, and confirmed that it looks good
### I replaced the REGISTRATION LINK text with our ajax form code block, saved, refreshed the preview, and now the form appears!
## unfortunately, the form is broken. clicking the "SUBSCRIBE" button (which, interestingly, looks different as it's inhereted the css from our osemain website) produces an error that the checkform() function is not defined
<pre>
ReferenceError: checkform is not defined[Learn More]
</pre>
### it appears to be unhappy about a syntax error in the middle of a comment. It looks like my multi-line comment tag is the html syntax, but I should really use the JS syntax. I changed it
## now it works better. I confirmed when I don't click the box "I agree to the OSE Privcacy Policy" that it pops-up with the error as expected. But when I do check the box, I get a pop-up "Sorry, we were unable to process your subscription." and the JS console shows that it's because of a Cross-Origin Request being blocked. Right, I need to add both www.opensourceecology.org _and_ microfactory.opensourceecology.org into the whitelist of phplist.opensourceecology.org's headers in the phplist config file
<pre>
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=3. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘https://microfactory.opensourceecology.org’).
</pre>
### well shit, it looks like the fucking Access-Control-Allow-Origin header only supports a single domain! The w3c recommends dynamically responding with the single approved domain using logic to determine who has requested the content and if they're in the whitelist. That sucks! https://www.w3.org/TR/cors/#resource-implementation
### fortunately, we can do this in nginx--which would occur before the varnish cache https://stackoverflow.com/questions/1653308/access-control-allow-origin-multiple-origin-domains/12414239#12414239
### I commented-out my line in the phplist config.php setting ACCESS_CONTROL_ALLOW_ORIGIN. This reset it back to "http://phplist.opensourceecology.org"
### I spent much time trying to get the nginx config to set the ACCESS_CONTROL_ALLOW_ORIGN header, but it never changed from "http://phplist.opensourceecology.org". Indeed, it appears that nginx won't set headers when using proxy_pass https://stackoverflow.com/questions/14501047/how-to-add-a-response-header-on-nginx-when-using-proxy-pass
### hmm, actually, for some reason the nginx config file that's applying for phplist is 'awstats.opensourceecology.org'. No wonder no changes I make are applying..
### doh! the filename was wong; it doesn't end in .conf!
<pre>
root@hetzner2 conf.d]# date
Sat Jan 26 12:26:26 UTC 2019
[root@hetzner2 conf.d]# pwd
/etc/nginx/conf.d
[root@hetzner2 conf.d]# ls -1 phplist*
phplist.opensourceecology.org
phplist.opensourceecology.org.4443.disabled
[root@hetzner2 conf.d]# ls -1 awstats*
awstats.openbuildinginstitute.org.conf
awstats.opensourceecology.org.conf
[root@hetzner2 conf.d]# 
</pre>
### I fixed this with the following nginx config
<pre>
[root@hetzner2 ~]# grep -A14 'location /' /etc/nginx/conf.d/phplist.opensourceecology.org.conf 

  location / {
	proxy_pass http://127.0.0.1:6081;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto https;
	proxy_set_header X-Forwarded-Port 443;
	proxy_set_header Host $host;

	# handle cors whitelist for ajax subscription to phplist
	proxy_hide_header Access-Control-Allow-Origin;
	if ( $http_origin ~ "^https://(www.opensourceecology.org|microfactory.opensourceecology.org)$" ) {
	  add_header Access-Control-Allow-Origin $http_origin;
	}

  }
</pre>
### I also updated the phplist config with a comment pointing out that this logic is now in nginx, not php
<pre>
[root@hetzner2 ~]# grep -B8 "define('ACCESS_CONTROL_ALLOW_ORIGIN" /var/www/html/phplist.opensourceecology.org/config.php  

// allow AJAX queries to add subscribers to our db from other domains
// Note: The ACCESS_CONTROL_ALLOW_ORIGIN header does not support multiple
//       domains, so we instead have to maintain a whitelist logically and
//       dynamically return the relevant domain iff it's in the whitelist.
//       Therefore, we actually override this phplist ACCESS_CONTROL_ALLOW_ORIGIN
//       header in our nginx config. See the relevant nginx config file:
//         * /etc/nginx/conf.d/phplist.opensourceecology.org.conf
#define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://www.opensourceecology.org" );
[root@hetzner2 ~]# 
</pre>
# It works! I made some further changes to it so that it looks good in the osemain wordpress post, and finally documented the final ajax form block on the wiki https://wiki.opensourceecology.org/wiki/Phplist#Ajax_Form
# I also documented the CORS stuff above https://wiki.opensourceecology.org/wiki/Phplist#Cross-Origin_Resource_Sharing
# I sent Marcin an email asking for review of the updated post & for a description of the Design Sprints list
# I updated the description of the OSEmal list with what Marcin provided in the blog post
<pre>
OSEmail is our main OSE Newsletter featuring news updates, workshop announcement, progress reports, and other noteworthy items. OSEmail comes out a few times per year at monthly or longer intervals. Anyone can sign up to receive our free newsletter. You can see more information at https://wiki.opensourceecology.org/wiki/OSEmail
</pre>
## This is now reflected in our main subscription signup page on phplist's site (the one that we'll link to for people who want to subscribe to the Design Sprints list) https://phplist.opensourceecology.org/lists/?p=subscribe&id=1

=Wed Jan 16, 2019=
# I logged into the server and found that the backup was currently uploading today's nightly backup to backblaze b2
# The logs are being hella spammed with the upload progress bar (the backup log containing just today's backup was already 14M), so I updated the backup.sh script to use the --noProgress argument.
<pre>
[root@hetzner2 ~]# du -sh /var/log/backups/*
6.2M    /var/log/backups/backup.log
48K     /var/log/backups/backup.log-20181206.gz
96K     /var/log/backups/backup.log-20181207.gz
48K     /var/log/backups/backup.log-20181209.gz
48K     /var/log/backups/backup.log-20181210.gz
96K     /var/log/backups/backup.log-20181211.gz
348K    /var/log/backups/backup.log-20181213.gz
1012K   /var/log/backups/backup.log-20181215.gz
340K    /var/log/backups/backup.log-20181216.gz
356K    /var/log/backups/backup.log-20181218.gz
748K    /var/log/backups/backup.log-20181219.gz
364K    /var/log/backups/backup.log-20181221.gz
372K    /var/log/backups/backup.log-20181222.gz
716K    /var/log/backups/backup.log-20181223.gz
712K    /var/log/backups/backup.log-20181225.gz
352K    /var/log/backups/backup.log-20181226.gz
392K    /var/log/backups/backup.log-20181227.gz
352K    /var/log/backups/backup.log-20181228.gz
352K    /var/log/backups/backup.log-20181230.gz
356K    /var/log/backups/backup.log-20181231.gz
360K    /var/log/backups/backup.log-20190101.gz
360K    /var/log/backups/backup.log-20190102.gz
700K    /var/log/backups/backup.log-20190103.gz
352K    /var/log/backups/backup.log-20190105.gz
352K    /var/log/backups/backup.log-20190106.gz
364K    /var/log/backups/backup.log-20190107.gz
388K    /var/log/backups/backup.log-20190108.gz
408K    /var/log/backups/backup.log-20190109.gz
360K    /var/log/backups/backup.log-20190110.gz
488K    /var/log/backups/backup.log-20190111.gz
352K    /var/log/backups/backup.log-20190113.gz
384K    /var/log/backups/backup.log-20190114.gz
14M     /var/log/backups/backup.log-20190115
[root@hetzner2 ~]# 
79M     .
[root@hetzner2 ~]# 
</pre>
# I also added many INFO echos to the script to make the logs more useful
# I ssh'd into our dreamhost server. Our ose_marcin user's home directly is currently using 304G of disk space. Hetzner1 backups are at 13G & hetzner2 backups are at 288G
<pre>
hancock% pwd
/home/marcin_ose
hancock% date
Wed Jan 16 02:04:22 PST 2019
hancock% du -sh .
304G	.
hancock% du -sh hetzner1
13G	hetzner1
hancock% du -sh hetzner2
288G	hetzner2
hancock% 
</pre>
# I confirmed that the backup from today (20190116) was not present on the dreamhost server. success!
<pre>
hancock% du -sh hetzner1/*
12G	hetzner1/20180501-052002
259M	hetzner1/20180502-052001
464M	hetzner1/20180602-052001
464M	hetzner1/20180702-052001
hancock% du -sh hetzner2/*
15G	hetzner2/20180501-072001
15G	hetzner2/20180601_072001
15G	hetzner2/20180701_072001
16G	hetzner2/20180801_072001
17G	hetzner2/20180901_072001
17G	hetzner2/20181001_072001
17G	hetzner2/20181101_072001
17G	hetzner2/20181202_072001
33G	hetzner2/20190101_072001
33G	hetzner2/20190112_072001
33G	hetzner2/20190113_072001
33G	hetzner2/20190114_072001
33G	hetzner2/20190115_072001
hancock% 
</pre>

=Tue Jan 15, 2019=

# I renamed the /root/backups/backup.sh script to be 'backup.old.20180115.sh' and I renamed backup2.sh to be 'backup.sh'
# I also renamed /etc/cron.d/backup_to_dreamhost to be 'backup_to_backblaze'
# The above has the effect of disabling backups being sent to dreamhost. All backups will now only go to backblaze b2 going forward.

=Thr Jan 10, 2019=

# checked backblaze wui
## total size of bucket is currently 157.7G
## I confirmed that th yearly backup file exists at 17.5G = yearly_hetzner2_20190101_111520.tar.gpg
## I checked our billing; in all of 2018 it was $0.02. In 2019, it's $0.63 so far in 2019. (which is actually the billing period between Dec 16 through Jan 10).
### I should do a test downloading the 2019-01-01 file and document the process (with screenshots) on our wiki after 2019-01-17--since we get a free quota of downloads once per month, and I used some of it for testing/documenting the download process via the cli in this billing period (which cost $0.61 to do)

=Wed Jan 09, 2019=

# marked time
# still waiting on Marcin to unblock me for phplist