Maltfield Log/2019 Q1
My work log from the year 2019 Quarter 1. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
Contents
- 1 See Also
- 2 Thr Mar 28, 2019
- 3 Sun Mar 24, 2019
- 4 Sat Mar 23, 2019
- 5 Thr Mar 21, 2019
- 6 Wed Mar 20, 2019
- 7 Tue Mar 19, 2019
- 8 Mon Mar 18, 2019
- 9 Sun Mar 17, 2019
- 10 Sat Mar 16, 2019
- 11 Fri Mar 15, 2019
- 12 Mon Feb 26, 2019
- 13 Sat Feb 24, 2019
- 14 Sat Feb 23, 2019
- 15 Sun Feb 17, 2019
- 16 Fri Feb 15, 2019
- 17 Tue Feb 12, 2019
- 18 Sat Feb 02, 2019
- 19 Tue Feb 05, 2019
- 20 Sat Jan 26, 2019
- 21 Wed Jan 16, 2019
- 22 Tue Jan 15, 2019
- 23 Thr Jan 10, 2019
- 24 Wed Jan 09, 2019
See Also
Thr Mar 28, 2019
- added updated gandi credentials to our shared ose keepass
- Marcin had issues sshing in again. I'll discuss it with him tomorrow during our phplist training call
- ...
- disabled the "invite" plugin so someone couldn't accidentally issue another repermission campaign
- ...
- trained Marcin on phplist
Sun Mar 24, 2019
- woocommerce & openbazaar research
- I need to look into coin gate & coinpayments.net plugins for woocommerce so we can take CC + crypto
Sat Mar 23, 2019
- woocommerce & openbazaar research
Thr Mar 21, 2019
- I checked our subscribe list status:
- Design Sprints= 53 (7, 300)
- True Fans = 93 (68, 856)
- OSEmail = 180 (12, 903)
- The phplist wui shows only 8 new bounces from march 20th, this is really, really low considering I sent it to 1,000 emails. Here's the relevant diagnostic codes for the bounces
- Diagnostic-Code: smtp; 554 delivery error: dd Sorry, your message to XYZ@yahoo.com cannot be delivered. This mailbox is disabled (554.30). - mta4343.mail.ne1.yahoo.com
- Diagnostic-Code: smtp; 552-5.2.2 The email account that you tried to reach is over quota. Please direct 552-5.2.2 the recipient to 552 5.2.2 https://support.google.com/mail/?p=OverQuotaPerm k13si1452811wrq.441 - gsmtp
- Diagnostic-Code: smtp; 550 5.1.1 <XYZ@shaw.ca> recipient rejected
- Diagnostic-Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser f2si1586770wrg.180 - gsmtp
- Diagnostic-Code: smtp; 554 5.7.1 <XYZ@pm.me>: Recipient address rejected: this address does not exist
- Diagnostic-Code: smtp; 522-5.7.1 <jared@hidden-knowledge.info>: Recipient address rejected: Requested mail action aborted: exceeded storage allocation 522 5.7.1 Please see http://support.mailhostbox.com/email-administrators-guide-error-codes/ for explanation of the problem.
- None of that appears to be due to a misconfigured domain/PTR record causing the reject due to possible spam or sth. Therefore, I requeued the campaign. I think that _should_ resend it to all the previously bounced mails.
- It sent to exctly one subscriber. Unfortuantely, this was a subscriber who I emailed with — they were a _new_ subscriber that I just blacklisted x_X
- So what I need to do is find the list of subscribers who've bounced and send only to them
- I found that the campaign's statistics listed the bounces, but without details. It shows that there were 37 bounces. I would _love_ to resend to those 37 people--my guess is that there's at least 10 that were bounced that wouldn't be bounced if we retired with the fixed PTR/hostnames
- In "System" -> "Manages" -> "View Bounces per List", I found there were 15 emails bounced from Design Sprints and 9 from OSEmail. I can get the list of emails from here, but it's unclear how to specify these specific emails for resending the campaign
- Ugh, after spending another hour researching this, I found it to be very non-trivial and I have to move forward. I just went ahead and published the wordpress post. Since [a] we hope this means we'll get more users to signup and [b] new subscribers would get blacklisted if we requeue the repermission campaign, then we won't be sending a repermission campaign again (unless I create a new campaign and a new list manually constructed of the list of bounced email addresses)
- I sent an email to Marcin asking to post a link to the new blog post on our social media https://www.opensourceecology.org/moving-to-open-source-email-list-software/
Wed Mar 20, 2019
- now that I think I've fixed the hostname/PTR issue, I want to re-send the re-permission campagn to only the email addresses that were bounced, but first I have to verify that requeuing the existing campaign won't re-blacklist old users that have already recieved the re-permission email
- I tested this by creating a new list called "justMe" — which only includes my email address as a single subscriber
- I sent a new invite campaign and sent it to myself. I got it. I then requeued it, and confirmed that I did not get a second email. I then cicked the confirmation link to unblacklist myself, and I requeued the campaign again. I confirmed again that I did not get a second email. Looks good.
- I edited the old re-permission campaign, setting the list only to "Design Sprints", and I put it in the queue again
- before putting it in the queue, I confirmed that the subscribe lists status are:
- Design Sprints= 52 (7, 301)
- True Fans = 93 (68, 856)
- OSEmail = 1,039 (12, 43)
- Afer requeueing the campaing, the lists changed
- Design Sprints = 52 (7, 301)
- True Fans = 93 (68, 856)
- OSEmail = 40 (12, 1,042)
- ugh, so it didn't blacklist any additional users in the Design Sprints or True Fans list, but it did send it out to OSEmail. It appears that somehow my edit included "all lists" *sigh* — I guess that's OK
Tue Mar 19, 2019
- Marcin had ssh issues "read: Connection reset by peer" — I noticed that iptables had a rule dropping in & out connections from an IP in Missouri. I restarted iptables and told Marcin to try again
- ...
- I checked our phplist campaigns. So far there's 87 confirmed users in True Fans and 47 confirmed users in Design Sprints.
- I checked the bounces; there were 19 more
- I got some issues with the use of the hostname "localhost" = "smtp; 504 5.5.2 <localhost>: Helo command rejected: need fully-qualified hostname"
- so, I think I'm going to have to change this to mail.opensourceecology.org and update all the PTR addresses. Again, this is so that [a] we don't get rejected when the FQDN PTR record (opensourceecology.org) differs from the hostname (previously hetzner2.opensourceecology.org) and [b] if I change the hostname to be just opensourceecology.org, then postfix will complain that the user does not exist (on the local system) instead of sending it out to the MTR record (we use google apps = gsuite)
- I tried to login to our "gandi" dns account, but the creds in our shared keepass file failed! I sent Marcin an email asking if he reset the password or sth
- then I realized that our opensourceecology.org dns is actually tied to our dreamhost account, which defines cloudflare as our nameserver
- I logged into our cloudflare account, and I realized that 'mail.opensourceecology.org' is already set to ghs.googlehosted.com
- That may or may not be necessary still. To be safe, I think I'll make this "mailer.opensourceecology.org"
- I created an A record for mailer.opensourceecology.org -> 138.201.84.243
- I created an AAAA record for mailer.opensourceecology.org -> 2a01:4f8:172:209e::2
- I logged into our hetzner2 "robot" wui console and changed the PTR "reverse DNS entry" to "mailer.opensourceecology.org" for 138.201.84.223, 138.201.84.243, and 2a01:4f8:172:209e::2
user@personal:~$ dig -x "2a01:4f8:172:209e::2" ; <<>> DiG 9.10.3-P4-Debian <<>> -x 2a01:4f8:172:209e::2 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41430 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.0.2.2.7.1.0.8.f.4.0.1.0.a.2.ip6.arpa. IN PTR ;; ANSWER SECTION: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.0.2.2.7.1.0.8.f.4.0.1.0.a.2.ip6.arpa. 7200 IN PTR mailer.opensourceecology.org. ;; Query time: 432 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) ;; WHEN: Tue Mar 19 13:04:08 EDT 2019 ;; MSG SIZE rcvd: 143 user@personal:~$ dig -x "138.201.84.243" ; <<>> DiG 9.10.3-P4-Debian <<>> -x 138.201.84.243 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21354 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;243.84.201.138.in-addr.arpa. IN PTR ;; ANSWER SECTION: 243.84.201.138.in-addr.arpa. 86400 IN PTR mailer.opensourceecology.org. ;; Query time: 352 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) ;; WHEN: Tue Mar 19 13:04:18 EDT 2019 ;; MSG SIZE rcvd: 98 user@personal:~$ dig -x "138.201.84.223" ; <<>> DiG 9.10.3-P4-Debian <<>> -x 138.201.84.223 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38500 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;223.84.201.138.in-addr.arpa. IN PTR ;; ANSWER SECTION: 223.84.201.138.in-addr.arpa. 86400 IN PTR mailer.opensourceecology.org. ;; Query time: 188 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) ;; WHEN: Tue Mar 19 13:04:33 EDT 2019 ;; MSG SIZE rcvd: 98 user@personal:~$
- I changed the "myhostname" value in /etc/postfix/main.cf to "mailer.opensourceecology.org"
- I sent a test campaign email. The new hostname was reflected in the source of the incoming email
Received: by 2002:a67:8e42:0:0:0:0:0 with SMTP id q63csp3982740vsd; ... X-Received: by 2002:adf:81a1:: with SMTP id 30mr19558287wra.285.1553015615877; ... Received: from mailer.opensourceecology.org (mailer.opensourceecology.org. [2a01:4f8:172:209e::2]) ... Received: by localhost (Postfix, from userid 48) ... Received: from localhost.localdomain [127.0.0.1] by phplist.opensourceecology.org with HTTP; Tue, 19 Mar 2019 17:13:35 +0000
- This confirms that
- [a] The hostname is no longer localhost in the source of the email (except at the end, which might be ok?)
- [b] The hostname is no longer "opensourceecology.org"
- [c] Emails can still be sent to "@opensourceecology.org" users who don't exist on the system
Mon Mar 18, 2019
- checked the stats on the re-permission campaign that I sent ~24 hours ago
- 49 out of 348 users have already clicked one of the links in this email.
- 42 out of 348 clicked the big green "I agree to the new Privacy Policy...Yes, keep sending me the OSEmail newsletter!" confirmation link. The rest are still blacklisted and will remain blacklisted unless the user actively clicks the confirmation link.
- we hadn't been recieving our campaign status emails from phplist@opensourceecology.org. I saw that this user was blacklisted, and it therefore may be an issue for recieving even stats emails. I unblacklisted the user by clicking the most recent confirmation link in the last repermission campaign that it recieved
- I didn't see any more bounces processed in the WUI
- I manually ran the command listed in the cron job, and it succesfully grabbed the 15 bounces from phplist_bounces@opensourceecology.org via POP. I confirmed that the bounce emails were archived in the gmail wui and that the bounces were visible in the phplist wui.
- further inspection shows that I had a superfluious "*" field in the cron line. I fixed it:
20 4 * * * root su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessbounces &>> /var/log/phplist/processBounces.log"
- I went ahead and edited the re-permission campaign and changed the list to "True Fans", put it in the queue, and waited. After <1 minute, the log spammed out that it sent 933 messages.
- I checked the munin graphs for spikes from sending the campaign yesterday
- There's an obvious spike in postfix from ~500 bytes/sec to ~80,000 bytes/sec
- There's a cooresponding spike in nice, and a decrease in idle for CPU. I don't think I'm niceing anything here, so I'm guessing phplist is good.
- load jumped to ~1.6 during the campaign send. No problem.
- "Logged-in users" spiked too could that be me logging in multiple times over ssh or some threadded phplist thing?
- There's a swap spike, and memory moved around a bit
- established connections didn't change much
- firewall throghput definitely spiked from ~70 packets/sec to ~450
- bits/sec spiked too from ~60k to ~20M
- there's a bit of a disk spike
- overall, our beefy server probably won't notice anything with our campaigns but I'll keep an eye on it for the next 2 repermission campaigns..
- ...
- I added the emailing logic to the Backup Report script (/root/backups/backupReport.sh). Unfortunately, it fails to mail me (michael@opensourceecology.org). I see an error on /var/log/maillog stating that the user "michael" does not exist.
- Well, damn, that's because the localhost isn't the mail server. How do I tell it to check the MX record for where to send the email, rather than expecting it to be localhost?
- I think this is a consequnce of me renaming the hostname from 'hetzner2.opensourceecology.org' to 'opensourceecology.org' to fix an email bounce issue with the phplist repermission campaign.
- I added the line "myhostname = localhost" to /etc/postfix/main.cf per https://devanswers.co/postfix-statusbounced-unknown-user-user/
- that fixed the issue! but do new mails sent by phplist appear to come from hostname = "localhost" ??
- well, yes and no. I sent an email, and it now has these two headers:
Received: from localhost (opensourceecology.org. [2a01:4f8:172:209e::2]) ... Received: from localhost.localdomain [127.0.0.1] by phplist.opensourceecology.org with HTTP; Mon, 18 Mar 2019 15:44:37 +0000
- so I think the proper solution here is to set "myhostname = mail.opensourceecology.org" in /etc/postfix.main.cf, and change the PTR address for all 3x of our IP addresses (2x IPv4 + 1x IPv6) to be "mail.opensourceecology.org"
- I'll hold-off on doing this unless I realize this complexity is necessary
- anyway, I added to our backup cron (/etc/cron.d/backup_to_backblaze) a line to run the nice'd backup reporting script once every month on the 3rd of the month at 04:20 AM
[root@opensourceecology cron.d]# cat backup_to_backblaze 20 07 * * * root time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log 20 04 03 * * root time /bin/nice /root/backups/backupReport.sh [root@opensourceecology cron.d]#
- I updated the list of emails to send the report to. it includes me, marcin, our ops account, and our backblaze@opensourceecology.org account
- I ran the backup report script
- I forwarded the resulting email in my inbox to Marcin informing him of this new script and asking him to confirm if he also recieved the report in his email inbox
Sun Mar 17, 2019
- My plan is to send the repermission campaing to the Design Sprints list, then the True Fans list, then the OSEmail list with increasing size (and risk) with each send. My fear is that I'll end up accidentally blacklisting someone twice if they're subscribed to multiple of thses lists. So I test this now.
- I created a new list called "test2" with just one subscriber in it
- I added thta new subscriber to the "test" list as well
- I sent a test repermission campaign to the "test" list. Both users recieved the email.
- Without the new subscriber clicking the confirmation link, I edited -> resent the repermission campaign to the "test2" list. The new subscriber did *not* recieve a duplicate email from the campaign :D
- I clicked the confirmation link for the new subscriber (removing them from the blacklist), and the new subscriber then got the "welcome" email
- I edited -> resent the repermission campaign to the "test2" list.
- The new subscriber did *not* recieve a duplicate email from the campaign :D
- And the new subscriber did not get re-blacklisted
- Conclusion: as long as I don't create a new campaign, and I just edit the existing campaign, change the list, and stick it in the queue again, phpist should do what I want/expect
- ...
- ok, I edited our old "Opt-In Action Required for OSE News and Updates" campaign = 22, changed the list from "test" to "Design Sprints" (with 349 subscribers), changed the embargo time to be in the present (not past), and added it to the queue!
- I initiated the sending, and--wow--it sent them all at once!
test1
phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com
phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - Maximum time for queue proce
ssing: 99999 [0.0022420000] (106)
phpList - Recently sent : 0
phpList - Started [0.0028750000] (109)
phpList - Sending in batches of 1 emails [0.0002610000] (110)
phpList - select id from phplist_message where status not in ("draft", "sent", "prepared", "suspended") and e
mbargo [0.0001760000] (111)
phpList - Processing has started, [0.0015480000] (114)
phpList - One campaign to process. [0.0014800000] (116)
phpList - sending of this campaign will stop, if it is still going in 46 days 6 hours 57 minutes 10 seconds [
0.0033430000] (133)
phpList - Processing campaign 22 [0.0022950000] (140)
phpList - Looking for subscribers [0.0041040000] (145)
phpList - User select query select distinct u.id from phplist_listuser as listuser
inner join phplist_user_user as u ON u.id = listuser.userid
inner join phplist_listmessage as listmessage ON listuser.listid = listmessage.listid
left join phplist_usermessage as um ON (um.messageid = 22 and um.userid = listuser.userid)
where
listmessage.messageid = 22
and listmessage.listid = listuser.listid
and u.id = listuser.userid
and um.userid IS NULL
and u.confirmed and !u.blacklisted and !u.disabled
[0.0015140000] (148)
phpList - Found them: 349 to process [0.0022960000] (152)
...
pList - Processed 349 out of 349 subscribers [0.0177960000] (22595)
phpList - It took 18 days 14 hours 52 minutes 55 seconds to send this message [0.0045030000] (22599)
phpList - Script stage: 5 [0.0467940000] (22616)
phpList - 348 messages sent in 24.89 seconds (50323 msgs/hr) [0.0004740000] (22619)
phpList - 1 invalid email addresses [0.0018850000] (22621)
phpList - Finished this run [0.0063290000] (22627)
- Within 10 minutes, I see that 3x of our users clicked the big green "I agree to the new Privacy Policy...Yes, keep sending me the OSEmail newsletter!" confirmation link. It appears to be working! :D
- I checked our phplist_bounces@opensourceecology.org inbox to see if we had any bounces, and we did. I saw messages indicating that the user does not exist or the mailbox has been disabled for the following domains:
- primus.ca
- opencrossroads.com
- hushmail.com
- ugcs.caltech.edu
- host mail.ugcs.caltech.edu[131.215.169.31] said: 554 5.7.1 Client host rejected: cannot find your hostname, [138.201.84.223] (in reply to RCPT TO command)
- yahoo.de
- unithorpe.org
- Host or domain name not found. Name service error for name=unithorpe.org type=AAAA: Host not found
- earthlink.net
- 550 IP 138.201.84.223 is blocked by EarthLink. Go to earthlink.net/block for details. (in reply to MAIL FROM command)
- aol.com
- findacceptance.be
- squarefruitenergy.com
- Host or domain name not found. Name service error for name=squarefruitenergy.com type=AAAA: Host not found
- michaelaltfield.net
- yahoo.com
- hotmail.com
- mail.ru
- note that the last 4x domains were for fake subscribers that I created to test bounces ☺
- the only thing that's really concerning in the above list is the caltech.edu & earthlink.net responses saying that our IP address was blocked or rejected
- earthlink uses CBL & SpamHaus to detect spam. The article on earthlink provides links to search for your IP in their DB. I did, and I confirmed that we (138.201.84.223) weren't in any of their lists https://support.earthlink.net/articles/email/email-blocked-by-earthlink.php
- https://www.abuseat.org/lookup.cgi
- https://www.spamhaus.org/query/ip/138.201.84.223
- I sent an email to earthlink.net asking for them to unblock our IP address, and I got an immediate reply telling me that it was unblocked and that it may be blocked again if we are discovered to be an open relay. Hmm.
- earthlink uses CBL & SpamHaus to detect spam. The article on earthlink provides links to search for your IP in their DB. I did, and I confirmed that we (138.201.84.223) weren't in any of their lists https://support.earthlink.net/articles/email/email-blocked-by-earthlink.php
Hello michael@opensourceecology.org Sun Mar 17 06:00:33 2019, We have removed the block on mail originating from 138.201.84.223, please allow 2-24 hours for normal email traffic to resume. Please understand that IPs previously removed from the EarthLink Correct Connect database may be re-added in the future should the server be discovered to again be open for relay. Regards, -- Earthlink Abuse Department blockedbyearthlink@abuse.earthlink.net http://www.earthlink.net/about/policies/use.faces
- just to be sure, I did a test to see if our server is an open relay. It said "Error: could not connect to server" of course, that's because of iptables blocking port 25 http://www.spamhelp.org/shopenrelay/shopenrelaytest.php
- I checked /etc/postfix/main.cf, and there does not appear to be a mynetworks restrictions. The firewall is sufficient to prevent us from being an open realy, but just to be safe I set mynetworks_style to "host".
mynetworks_style = host
- Caltech seems to be because of a missing Reverse DNS = PTR lookup. I encountered this in the past https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q3#Thr_Jul_12.2C_2018
- so the ipv6 address from before looks OK
- Caltech seems to be because of a missing Reverse DNS = PTR lookup. I encountered this in the past https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q3#Thr_Jul_12.2C_2018
user@ose:~$ dig -x "2a01:4f8:172:209e::2" ; <<>> DiG 9.10.3-P4-Debian <<>> -x 2a01:4f8:172:209e::2 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 45913 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.0.2.2.7.1.0.8.f.4.0.1.0.a.2.ip6.arpa. IN PTR ;; Query time: 232 msec ;; SERVER: 10.139.1.2#53(10.139.1.2) ;; WHEN: Sun Mar 17 06:19:55 EDT 2019 ;; MSG SIZE rcvd: 101 user@ose:~$
- the PTR records eem to be fine on our 2x ipv4 addresses as well, though..
user@ose:~$ dig -x 138.201.84.243 ; <<>> DiG 9.10.3-P4-Debian <<>> -x 138.201.84.243 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37153 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;243.84.201.138.in-addr.arpa. IN PTR ;; ANSWER SECTION: 243.84.201.138.in-addr.arpa. 86400 IN PTR opensourceecology.org. ;; Query time: 226 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) ;; WHEN: Sun Mar 17 06:20:44 EDT 2019 ;; MSG SIZE rcvd: 91 user@ose:~$ dig -x 138.201.84.223 ; <<>> DiG 9.10.3-P4-Debian <<>> -x 138.201.84.223 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5214 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;223.84.201.138.in-addr.arpa. IN PTR ;; ANSWER SECTION: 223.84.201.138.in-addr.arpa. 86400 IN PTR opensourceecology.org. ;; Query time: 229 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) ;; WHEN: Sun Mar 17 06:20:49 EDT 2019 ;; MSG SIZE rcvd: 91 user@ose:~$
- ok, so maybe this actually isn't in regard to PTR records. Maybe I'm just missing the "hostname" field header in the email or something?
- googling this further points to PTR. But the PTR record says "opensourceecology.org." while the server itself is "hetzner2.opensourceecology.org"
- the quickest & lowest risk was to just change the server's hostname from "hetzner2.opensourceecology.org" to "opensourceecology.org" Ill see if that causes any issues or prevents further issues in the future. Note that this is not the scaleable solution, but unfortunately OSE doesn't have any IT budget anyway. If we ever think we'll have >2 prod servers in the future, this will need to be re-addressed. Certainly, it would be nice to have a dedicated mail server one day, and then we can name it 'mail.opensourceecology.org'.
[root@hetzner2 postfix]# hostname opensourceecology.org [root@hetzner2 postfix]# hostname opensourceecology.org [root@hetzner2 postfix]# cat /etc/hostname opensourceecology.org [root@hetzner2 postfix]#
- I got another bounce from isis.vanderbilt.edu due to an infinite mail loop (hop count exceeded). Interesting. Probably an issue on their end?
- oh, I've also confirmed that the bounce email's filter to prevent incoming mail from being marked as spam _does_ work as desired
- I see that 3x more people have opened our email. 2/3 have clicked te confirm link. So 1 that has opened it still remains blacklisted.
- I found a very nice summary of the campaign, which shows a list of subscribers that have opened the campaign. It also shows the status (green thumbs-up or red thumbs-down) to show who has clicked the big green button to un-blacklist themselves https://phplist.opensourceecology.org/lists/admin/?page=userclicks&msgid=22
- ********** this is accessible via Statistics -> Campaign Click Statistics -> Campaign -> View Subscribers (yeah, it's pretty buried)
- oh, that's actually not the best summary; it only shows users who *did* click. The reason that the one guy who is blacklisted is listed is because they actually clicked the "unsubscribe" link
- ...
- Marcin still hasn't confirmed that he was able to login to the backup b2 account to validate the contents of our bucket to include recent backupsl. This is insufficient; we need a system to report on error
- I'd like to have a system to check recent backups and report only on error, but what happens if that system itself fails and no emails come through? I think the best thing is to just send a (bi?) monthly report that lists the contents of the B2 bucket. It should do a few basic sanity checks and either say "ALL GOOD" or "WARNING: X BACKUP MISSING" and send it out to a set of email addresses
- I hacked up this script's logic for checking the b2 bucket contents it just needs to have the actual body of the email typed up & the sending of the email.
Sat Mar 16, 2019
- I did a quick search on the phplist forums about the campaign sending throttle options
- MAX_PROCESS_MESSAGE
- MAILQUEUE_BATCH_SIZE
- MAILQUEUE_BATCH_PERIOD
- MAILQUEUE_THROTTLE
- MAILQUEUE_AUTOTHROTTLE
- USE_DOMAIN_THROTTLE
- DOMAIN_BATCH_SIZE
- DOMAIN_BATCH_PERIOD
- DOMAIN_AUTO_THROTTLE
- MAX_PROCESSQUEUE_TIME
- Most of what I saw was in regard to issues with user's hosting companies throttling them on a shared server. That's not relevant.
- I did see some stuff about gmail and domain throttling, but the answer there was to just throttle after issues and resend. But what about bounce management?
- I found a doc on bounce management. it is very detailed and great, and is making me more aware of how much pain I may be in once we begin sending mass emails in our repermission campaign x_X https://www.phplist.org/manual/ch040_bounce-management.xhtml
- ok, so the above documentation explains why attempting to process bounces fails!
[root@hetzner2 cron.d]# su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessbounces" test1 phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - Cannot create POP3 connection to localhost: Connection failed to localhost.localdomain,110: Connection refused phpList - Download failed, exiting real 0m0.138s user 0m0.058s sys 0m0.061s [root@hetzner2 cron.d]#
- per the defaults, we have it setup to use pop. we should change this to mbox since we're on a dedicated server
[root@hetzner2 phplist.opensourceecology.org]# grep -A 56 'Message envelope.' config.php
// Message envelope.
// This is the address that most bounces will be delivered to
// Your should make this an address that no PERSON reads
// but a mailbox that phpList can empty every so often, to process the bounces
// $message_envelope = 'listbounces@yourdomain';
// Handling bounces. Check README.bounces for more info
// This can be 'pop' or 'mbox'
$bounce_protocol = 'pop';
// set this to 0, if you set up a cron to download bounces regularly by using the
// commandline option. If this is 0, users cannot run the page from the web
// frontend. Read README.commandline to find out how to set it up on the
// commandline
define('MANUALLY_PROCESS_BOUNCES', 1);
// we set MANUALLY_PROCESS_QUEUE to 0 so that the "Process the Queue" button
// will disappear from the WUI as the queue is processed via cron
define('MANUALLY_PROCESS_QUEUE', 0);
// whitelist of users that are permitted to execute bin/phplist for cli control
$commandline_users = array( "apache" );
// when the protocol is pop, specify these three
$bounce_mailbox_host = 'localhost';
$bounce_mailbox_user = 'popuser';
$bounce_mailbox_password = 'password';
// the "port" is the remote port of the connection to retrieve the emails
// the default should be fine but if it doesn't work, you can try the second
// one. To do that, add a # before the first line and take off the one before the
// second line
$bounce_mailbox_port = '110/pop3/notls';
//$bounce_mailbox_port = "110/pop3";
// it's getting more common to have secure connections, in which case you probably want to use
//$bounce_mailbox_port = "995/pop3/ssl/novalidate-cert";
// when the protocol is mbox specify this one
// it needs to be a local file in mbox format, accessible to your webserver user
$bounce_mailbox = '/var/mail/listbounces';
// set this to 0 if you want to keep your messages in the mailbox. this is potentially
// a problem, because bounces will be counted multiple times, so only do this if you are
// testing things.
$bounce_mailbox_purge = 1;
// set this to 0 if you want to keep unprocessed messages in the mailbox. Unprocessed
// messages are messages that could not be matched with a user in the system
// messages are still downloaded into phpList, so it is safe to delete them from
// the mailbox and view them in phpList
$bounce_mailbox_purge_unprocessed = 1;;
// how many bounces in a row need to have occurred for a user to be marked unconfirmed
$bounce_unsubscribe_threshold = 5;
[root@hetzner2 phplist.opensourceecology.org]#
- I also went to "System" -> "Manage bounces" in the WUI. It said that I had no bounce rules defined, and I saw no bounces in the list
You currently have no rules defined. You can click Generate Bounce Rules in order to auto-generate rules from your existing bounces. This will results in a lot of rules which you will need to review and activate. It will however, not catch every single bounce, so it will be necessary to add new rules over time when new bounces come in.
- so I'm not sure that I will be able to predict what bounce rules I want before our first campaign, but I do think it's important that I can record bounces and view them in the WUI.
- oh shit, I guess that since we use Gsuite/Google Apps for email (MX records on opensourceecology.org to google's servers), we actually can't use an mbox format for checking bounces. So I imagine we can use pop on gmail servers.
- Or maybe we could use bounces@phplist.opensourcecology.org or something--such that the subdomain had a distinct MX record pointing to our server. I'm not sure if that's how MX records work, though.
- So the easist/best solution is to just create a new email account on Gsuite for collecting bounces. My biggest concern here is that POP is totally insecure. I need to make sure it's TLS'd.
- I found the relevant option, supported by this guide that specifically says how to use a gmail address for bounces https://www.inmotionhosting.com/support/edu/phplist/301-process-bounces-from-gmail-in-phplist
$bounce_mailbox_port = "995/pop3/ssl";
- note that our current config (by default) will mark a user as unconfirmed if we recieve 5 bounces in-a-row
- I updated the relevant config options:
- $bounc_mailbox_host
- $bounc_mailbox_user
- $bounc_mailbox_password
- $bounc_mailbox_port
- now the run tries to connect, but fails
[root@hetzner2 cron.d]# su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessbounces" test1 phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - Cannot create POP3 connection to pop.gmail.com: POP3 connection broken in response phpList - Download failed, exiting real 0m10.181s user 0m0.060s sys 0m0.064s [root@hetzner2 cron.d]#
- The failure is probably because I have iptables configured not to allow apache to make requests to the internet, effectively cutting the legs off of most hacks. This is evident in the iptables log
[root@hetzner2 phplist.opensourceecology.org]# tail -f /var/log/kern.log Mar 16 09:28:32 hetzner2 kernel: iptables OUT denied: IN= OUT=eth0 SRC=138.201.84.223 DST=2.16.186.27 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27866 DF PROTO=TCP SPT=56468 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 ...
- the above ip address is actually akamai, not google. hmm.
- actually, when I dig deeper at the iptables config, I see that the log line occurs before the drop line. I assume that's because the drops were just too damn high (and flooding log files). Whatever.
- I commented-out the iptables rule blocking outgoing ports for apache & reloaded, but I got the same output. damn.
[root@hetzner2 cron.d]# su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessbounces" test1 phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - Cannot create POP3 connection to pop.gmail.com: POP3 connection broken in response phpList - Download failed, exiting real 0m10.187s user 0m0.052s sys 0m0.073s [root@hetzner2 cron.d]#
- ah, right, I forgot that pop is disabled for new gmail accounts by default. I logged in as this user and enabled POP. I also set it to archive emails after being downloaded via pop.
- still no good
[root@hetzner2 cron.d]# su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessbounces" test1 phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - Cannot create POP3 connection to pop.gmail.com: POP3 connection broken in response phpList - Download failed, exiting real 0m10.175s user 0m0.052s sys 0m0.073s [root@hetzner2 cron.d]#
- ugh, I changed the port line from 'tls' to 'ssl', and it worked. It's not ideal, but hopefully it's secure enough.
$bounce_mailbox_port = '995/pop3/ssl';
[root@hetzner2 cron.d]# su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessbounces" test1 phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - 2 bounces to fetch from the mailbox phpList - Please do not interrupt this process phpList - UID0 MSGID0 phpList - Deleting message 1 phpList - UID0 MSGID0 phpList - Deleting message 2 phpList - Closing mailbox, and purging messages phpList - reprocessing phpList - 2 bounces to reprocess phpList - 2 out of 2 processed phpList - 0 bounces were re-processed and 0 bounces were re-identified phpList - Identifying consecutive bounces phpList - Nothing to do phpList - total of 0 subscribers processed real 0m3.898s user 0m0.064s sys 0m0.075s [root@hetzner2 cron.d]#
- after the above run, I confirmed that the 2x "welcome" emails in the inbox were no longer in the inbox, but they _are_ visible in the "All Mail" section
- To test it, I added a new subscriber = thisisnotarealuser@opensourceecology.org to the "test" list. I created a new campaign and sent it to the "test" list.
- after the campaign was sent, I didn't get an email in the phplist_bounces@opensourceecology.org mailbox. I checked the one that _did_ arrive in my mailbox, and I couldn't find any message envelope header!
- ah, but I got a "Bounces-To:" header defined with the address above. And spf-related headers added by Google's servers mentioning this emal address
- ahh, it didn't even send it to the fake user since they weren't confirmed. I manually marked them as "confirmed" and sent the campaign again. Note that since I edited the old campaign, it only sent one mail this time since it already sent it to the old user and editing an old campaign wont' send the same campaign to the same subscriber >1 time.
- unfortunately, I still didn't get any bounce mails in the inbox for phplist_bounces@opensourceecology.org
- I verfied that the email address in the header sent in the email campaign exactly matched the inbox's email address. there's no typo here.
- ah, they came! 4 "Undelivered Mail Returned to Sender" emails were in the spam folder :D
- for some reason they all came from MAILER-DAEMON@hetzner2.opensourceecology.org
- I ran the script to process the bounces
==> /var/log/phplist/processBounces.log <== test1 phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - 0 bounces to fetch from the mailbox phpList - reprocessing phpList - 2 bounces to reprocess phpList - 2 out of 2 processed phpList - 0 bounces were re-processed and 0 bounces were re-identified phpList - Identifying consecutive bounces phpList - Nothing to do phpList - total of 0 subscribers processed
- unfortunately, when I go to "System" -> "Manage bounces" -> "Unidentified" in the phplist WUI, I only see the first 2x welcome messages. the emails in the spam folder are still marked as unread. I guess
- I created a pretty dumb filter that says "is:spam" then "never send it to Spam" https://webapps.stackexchange.com/questions/69442/how-to-disable-gmail-anti-spam-completely
- I manually moved the existing 4 spam messages to the inbox by marking them as "not spam", then I re-ran the processbounces
test1 phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - 4 bounces to fetch from the mailbox phpList - Please do not interrupt this process phpList - UID2384 MSGID27 phpList - Deleting message 1 phpList - UID2386 MSGID27 phpList - Deleting message 2 phpList - UID2382 MSGID27 phpList - Deleting message 3 phpList - UID2383 MSGID27 phpList - Deleting message 4 phpList - Closing mailbox, and purging messages phpList - reprocessing phpList - 2 bounces to reprocess phpList - 2 out of 2 processed phpList - 0 bounces were re-processed and 0 bounces were re-identified phpList - Identifying consecutive bounces phpList - total of 4 subscribers processed
- now, when I go to "System" -> "Manage bounces" -> "Processed" in the phplist WUI, I see the 4x mails, and they've been correctly linked to the appropriate subscriber in the phplist system. Awesome!
- while I'm not entirely confident that I've solved the spam-marking issue, I am happily convinced that bounces are being correctly recieved, and not simply going into an internet black-hole. That's a sufficient milesone for beginning the repermission campaign!
- I restored the iptables rules preventing apache user from reaching out, and--for some reason--it still worked. Maybe iptables allows it because root is the parent process? hmm.
- I also documented this config as reasonably as I should without risking documentation rot https://wiki.opensourceecology.org/wiki/Phplist#Important_Files_.26_Directories
- ...
- I booked my time and logged my logs
- ...
- I also updated our documentation on backups https://wiki.opensourceecology.org/wiki/OSE_Server#Backups
- I realized that our amazon account is almost 1 year old. Indeed, the Dreamhost backup catastrophy occured in 2018-03, and it's now 2019-03. We have an old Janus server running in ec2 that's free for the first 12 months, but we'll be charged for it soon. So I went ahead and deleted it.
- I noticed that our cards were expired as well. I emailed Marcin to update the info; currently we're just paying $1.14 to retain old backup data from hetzner1 in Glacier.
- ...
- I went ahead and looked through our awstats & munin graphs for the first time in many months
- our wiki is getting ~1k unique visitors per day, ~ 26,000 hits per day
- osemain is getting ~800 unique visitors per day, ~ 22,300 hits per day
- munin shows varnish returning an average of 83% cach hits with 15k objects in the cahce
- munin shows the load at 0.15 avg. 0.01 min - 1.98 max. The spikes are likely the nice'd nightly backups
- munin does show a tiny spike in postfix byte throughput during my little campaign tests that'll be a good stat to track forphplist s
Fri Mar 15, 2019
- I created a new campaign and put it into the queue
- I ran the cli command to process the queue, and it worked
[maltfield@hetzner2 ~]$ sudo su -s /bin/bash apache -c "/usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -p processqueue"
[sudo] password for maltfield:
test1
phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com
phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - Maximum time for queue processing: 99999 [0.0043790000] (107)
phpList - Recently sent : 0
phpList - Started [0.0056810000] (110)
phpList - Sending in batches of 1 emails [0.0002560000] (111)
phpList - select id from phplist_message where status not in ("draft", "sent", "prepared", "suspended") and embargo [0.0002010000] (112)
phpList - Processing has started, [0.0033270000] (115)
phpList - One campaign to process. [0.0032120000] (117)
phpList - sending of this campaign will stop, if it is still going in 6 days 23 hours 27 minutes 34 seconds [0.0038360000] (134)
phpList - Processing campaign 24 [0.0125830000] (151)
phpList - Looking for subscribers [0.0104980000] (156)
phpList - User select query select distinct u.id from phplist_listuser as listuser
inner join phplist_user_user as u ON u.id = listuser.userid
inner join phplist_listmessage as listmessage ON listuser.listid = listmessage.listid
left join phplist_usermessage as um ON (um.messageid = 24 and um.userid = listuser.userid)
where
listmessage.messageid = 24
and listmessage.listid = listuser.listid
and u.id = listuser.userid
and um.userid IS NULL
and u.confirmed and !u.blacklisted and !u.disabled
[0.0021030000] (159)
phpList - Found them: 2 to process [0.0033930000] (163)
phpList - Sending 24 to catarinamfmota@gmail.com [0.0115400000] (174)
phpList - It took 0.0534280000 seconds to send [0.0563310000] (244)
phpList - Sending 24 to michael@opensourceecology.org [0.0233820000] (257)
phpList - It took 0.0307020000 seconds to send [0.0338240000] (299)
phpList - Processed 2 out of 2 subscribers [0.0185900000] (307)
phpList - It took very little time to send this message [0.0124160000] (322)
phpList - Script stage: 5 [0.0151830000] (331)
phpList - 2 messages sent in 0.22 seconds (32613 msgs/hr) [0.0003270000] (334)
phpList - Finished this run [0.0071450000] (340)
[maltfield@hetzner2 ~]$
- that confirms that the queue _can_ be processed by the cli. time for a cron job and similar confirmation!
- I created a new cron job at /etc/cron.d/phplist and associated log dir + logrotate config
- per the documentation, I set it up to process the queue once every 5 minutes and process the bounses once per day https://www.phplist.org/manual/ch036_setting-up-your-cron.xhtml
[root@hetzner2 cron.d]# mkdir /var/log/phplist [root@hetzner2 cron.d]# chown apache:apache /var/log/phplist [root@hetzner2 cron.d]# cat /etc/cron.d/phplist 0-59/5 * * * * root su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessqueue &>> /var/log/phplist/processQueue.log" 20 4 * * * * root su -s /bin/sh apache -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessbounces &>> /var/log/phplist/processBounces.log" [root@hetzner2 cron.d]#
- I created a new campaign, and I confirmed that it was sucessfully sent without me doing anything; the cron job picked it up on the 5-minute interval. great!
- the above test, of course, was only sent to the "test" list with 3 members. I'm now confident that a huge campaign will automatically process without user interverntion, but will the first wave be too large? There's no way to test it, but I should probably look into pre-emptively tweaking the batch size options. I found a few relevant options in the config_extended.php file
- MAX_PROCESS_MESSAGE
- MAILQUEUE_BATCH_SIZE
- MAILQUEUE_BATCH_PERIOD
- MAILQUEUE_THROTTLE
- MAILQUEUE_AUTOTHROTTLE
- USE_DOMAIN_THROTTLE
- DOMAIN_BATCH_SIZE
- DOMAIN_BATCH_PERIOD
- DOMAIN_AUTO_THROTTLE
- MAX_PROCESSQUEUE_TIME
Mon Feb 26, 2019
- Duncan Cameron responded to my question about using the phplist cli, suggesting that it might be just a permissions problem with the session files https://discuss.phplist.org/t/phplist-cli-p-processqueue-does-nothing-fails-at-languages-php-session-start/5025/2?u=maltfield
- I verified that the permissions were OK, but pointed out that I do have a hardened php session config, which may be an issue if--for example--there's some poorly designed code somewhere that expects the session name to be the default = PHPSESSID"
- I did some more digging. The lines that we're hitting this issue on is "@session_start()"
- I found that the "@" at the start of the function call causes erros to be suppressed. Stupid but true. I removed it, and I got some output
[maltfield@hetzner2 ~]$ sudo su -s /bin/bash apache -c "/usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -p processqueue"
[sudo] password for maltfield:
PHP Fatal error: session_start(): Failed to initialize storage module: files (path: ) in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php on line 57
PHP Stack trace:
PHP 1. {main}() /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php:0
PHP 2. include_once() /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php:102
PHP 3. session_start() /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php:57
before languages
test
[maltfield@hetzner2 ~]$
- note the "before lanugages" & "test" lines were echos that I added to determine where the issue lies
- I did some googling for "session_start()" and "failed to initialize storage module"
- I wonder if all the php.ini configs are applied when calling php via the cli. I checked it by trying to run a session_start() manually via the cli, and I got totally different results
[maltfield@hetzner2 ~]$ echo '<?php session_start();?>' | sudo su -s /bin/bash apache -c "/usr/bin/php"
PHP Fatal error: Call to undefined function ecsession_start() in - on line 1
PHP Stack trace:
PHP 1. {main}() -:0
[maltfield@hetzner2 ~]$ echo '<?php session_start();?>' | sudo su -s /bin/bash apache -c "/usr/bin/php"
PHP Warning: session_start(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/staging.opensourceecology.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/var/www/html/d3d.opensourceecology.org:/var/www/html/3dp.opensourceecology.org:/var/www/html/microfactory.opensourceecology.org:/var/www/html/phplist.opensourceecology.org:/var/www/html/microfactory.opensourceecology.org:/usr/share/cacti/:/etc/cacti/) in - on line 1
PHP Stack trace:
PHP 1. {main}() -:0
PHP 2. session_start() -:1
PHP Fatal error: session_start(): Failed to initialize storage module: files (path: ) in - on line 1
PHP Stack trace:
PHP 1. {main}() -:0
PHP 2. session_start() -:1
[maltfield@hetzner2 ~]$
- so, actually, it's trying to get to the session dir in "/tmp" that tells me it's veering off from the php.ini settings. But it still gets the open_basedir config from php.ini! Strange..
- it looks like I actually don't set "session.save_path" in php.ini. Instead, I use session.cookie_secure. I think that this config prevents sessions from being used over http, requiring https. That makes sense
- actually, that looks like a total config failure. The "session.cookie_secure" option should simply be a boolean http://php.net/manual/en/session.configuration.php#ini.session.cookie-secure
- I changed "session.cookie_secure" to "1" and set "session.save_path" to "/var/lib/php/session"
session.save_path = "/var/lib/php/session"
; Whether to use cookies.
; http://php.net/session.use-cookies
session.use_cookies = 1
; http://php.net/session.cookie-secure
;session.cookie_secure =
session.cookie_secure = 1
- now it works!
[maltfield@hetzner2 ~]$ sudo su -s /bin/bash apache -c "/usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -p processqueue"
test1
phpList - phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.com
phpList version 3.3.3 (c) 2000-2019 phpList Ltd, https://www.phplist.comphpList - Maximum time for queue processing: 99999 [0.0026350000] (107)
phpList - Recently sent : 0
phpList - Started [0.0042370000] (110)
phpList - Sending in batches of 1 emails [0.0002000000] (111)
phpList - select id from phplist_message where status not in ("draft", "sent", "prepared", "suspended") and embargo [0.0006080000] (112)
phpList - Processing has started, [0.0016170000] (115)
phpList - One campaign to process. [0.0015100000] (117)
phpList - sending of this campaign will stop, if it is still going in 3 days 22 hours 11 seconds [0.0035370000] (133)
phpList - Processing campaign 22 [0.0096130000] (150)
phpList - Looking for subscribers [0.0059470000] (155)
phpList - User select query select distinct u.id from phplist_listuser as listuser
inner join phplist_user_user as u ON u.id = listuser.userid
inner join phplist_listmessage as listmessage ON listuser.listid = listmessage.listid
left join phplist_usermessage as um ON (um.messageid = 22 and um.userid = listuser.userid)
where
listmessage.messageid = 22
and listmessage.listid = listuser.listid
and u.id = listuser.userid
and um.userid IS NULL
and u.confirmed and !u.blacklisted and !u.disabled
[0.0017760000] (158)
phpList - Found them: 0 to process [0.0027430000] (162)
phpList - Processed 0 out of 0 subscribers [0.0028240000] (165)
phpList - Hmmm, No subscribers found to send to [0.0014980000] (167)
phpList - It took very little time to send this message [0.0073380000] (183)
phpList - Script stage: 3 [0.0015600000] (186)
phpList - Finished, Nothing to do [0.0001340000] (187)
phpList - Finished, All done [0.0025960000] (192)
[maltfield@hetzner2 ~]$
- I did some research about why editing the campaign & sending it again doesn't actually send anything. It's because phplist won't send a campaign (even after being edited & resent) to a subscirber >1 time. https://discuss.phplist.org/t/requeue-send-message-again-to-same-people-or/1259/3
- this is very helpful information; it means that I can try out sending the campain to our smallest list, then re-send it to the next largets list, and then finally to OSEmail without fear that I'll send the campaign more than once to a subscriber who is subscribed to more than one list!
Sat Feb 24, 2019
- I traced the issue with the cli execution producing no output to the languates.php file's session_start() call https://github.com/phpList/phplist3/blob/849a53f8c93eb9940ae08180b960bd7a71957df2/public_html/lists/admin/languages.php#L56
- The only other reference to this issue I could find was on the old phplist forms from 2012. Unfortunately, it got no replies https://discuss.phplist.org/t/phplist-cli-p-processqueue-does-nothing-fails-at-languages-php-session-start/5025
- I posted a question about this to the phplist forums https://discuss.phplist.org/t/phplist-cli-p-processqueue-does-nothing-fails-at-languages-php-session-start/5025
Sat Feb 23, 2019
- logging time
- ...
- I sent a test repermission campagin to the 'test' list. It worked.
- One of my concerns about this real repermission campaign is sending mass mail for the first time. Will it work? Will it take hours? Days?
- Our OSEmail list has 1,061 subscribers. True Fans is 957. Design Sprints is 349. Therefore, I think we should first do a test of the repermission campaign with the Design Sprints. Then True Fans. THen OSEmail.
- I want to get cron queue processing complete before the real test, so that it will send the campaign to all our subscribers without me having to leave my web browser open.
- This documentation talks about sending campaigns via the wui, cli, cron, or script https://resources.phplist.com/documentation/sendingcampaign
- This documentation talks about cli options for batch processing https://resources.phplist.com/system/batch_processing
- I found the relevant binary for manipulating phplist via the cli on our server
[root@hetzner2 phplist.opensourceecology.org]# pwd /var/www/html/phplist.opensourceecology.org [root@hetzner2 phplist.opensourceecology.org]# ls -lah bin/phplist ----r----- 1 not-apache apache 766 May 15 2018 bin/phplist [root@hetzner2 phplist.opensourceecology.org]#
- It's important to note that this file is not executable. I fixed that here:
[root@hetzner2 phplist.opensourceecology.org]# chmod 0050 bin/phplist [root@hetzner2 phplist.opensourceecology.org]# ls -lah bin/phplist ----r-x--- 1 not-apache apache 766 May 15 2018 bin/phplist [root@hetzner2 phplist.opensourceecology.org]# pwd /var/www/html/phplist.opensourceecology.org [root@hetzner2 phplist.opensourceecology.org]#
- And we certainly don't want it to be executed as root for security reasons, so here's the command to execute it as the apache user. Interesting to note how it just spat a cronjob example at us when we use the '-pprocessqueue' argument per the documentation above
[maltfield@hetzner2 ~]$ sudo su -s /bin/sh apache -c "/usr/bin/php /var/www/html/phplist.opensourceecology.org/bin/phplist -pprocessqueue" # script to run phpList from commandline. You may need to edit this to make it work # with your shell environment. The following should work for Bash on Linux # but this may vary strongly in other situations. You will need to dig into the # code to make sure it works for you. # in commandline mode, access is restricted to users who are listed in the config file # check README.commandline for more info # # when you set this file up, and place it in your PATH, you can do eg # $ phplist -pprocessqueue # to run the queue # run the phpList index file with all parameters passed to this script # make sure the php binary is "cli" # /usr/bin/php /home/website/public_html/lists/admin/index.php -c /home/website/public_html/lists/config/config.php $* [maltfield@hetzner2 ~]$
- Hmm...It looks like after I sent the campaign to the "test" list, it was removed from the list of campagin drafts in Campaigns -> Send a campaign -> Draft campaigns
- I found that I could move an old campaign back into the "Draft campaigns" list by going to Campaigns -> List of campaigns -> View -> Edit campaign
- I re-sent the campaign to the 'test' list again, but this time I didn't process the queue in the wui. I tried to run the above command to process the queue on the cli, but it just ouputed the cron thingy again. Reading it, I realized that I hadn't specified any users who can run the script. I checked the config_extended.php file and extracted some relevant options
// If you set up your system to send the message automatically (from commandline),
// you can set this value to 0, so "Process Queue" will disappear from the site
// this will also stop users from loading the page on the web frontend, so you will
// have to make sure that you run the queue from the commandline
// check README.commandline how to do this
define('MANUALLY_PROCESS_QUEUE', 1);
...
// set this to 0, if you set up a cron to download bounces regularly by using the
// commandline option. If this is 0, users cannot run the page from the web
// frontend. Read README.commandline to find out how to set it up on the
// commandline
define('MANUALLY_PROCESS_BOUNCES', 1);
...
// if you use commandline, you will need to identify the users who are allowed to run
// the script. See README.commandline for more info
// $commandline_users = array("admin");
// or you can use the following to disable the check (take off the # in front of the line)
$commandline_users = array();
- I added the following to our config file
// we set MANUALLY_PROCESS_QUEUE to 0 so that the "Process the Queue" button
// will disappear from the WUI as the queue is processed via cron
define('MANUALLY_PROCESS_QUEUE', 0);
// whitelist of users that are permitted to execute bin/phplist for cli control
$commandline_users = array( "apache" );
- so the cron job actually tells me to use index.php. I tried it, but I'm still having the same problem..
[maltfield@hetzner2 ~]$ sudo su -s /bin/sh apache -c "/usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessqueue" [maltfield@hetzner2 ~]$
- oh, the contents of that phplist script is simply that cron job!
[root@hetzner2 phplist.opensourceecology.org]# date Sat Feb 23 18:53:58 UTC 2019 [root@hetzner2 phplist.opensourceecology.org]# pwd /var/www/html/phplist.opensourceecology.org [root@hetzner2 phplist.opensourceecology.org]# cat bin/phplist #!/bin/bash # script to run phpList from commandline. You may need to edit this to make it work # with your shell environment. The following should work for Bash on Linux # but this may vary strongly in other situations. You will need to dig into the # code to make sure it works for you. # in commandline mode, access is restricted to users who are listed in the config file # check README.commandline for more info # # when you set this file up, and place it in your PATH, you can do eg # $ phplist -pprocessqueue # to run the queue # run the phpList index file with all parameters passed to this script # make sure the php binary is "cli" # /usr/bin/php /home/website/public_html/lists/admin/index.php -c /home/website/public_html/lists/config/config.php $* [root@hetzner2 phplist.opensourceecology.org]#
- executing the command does nothing, so I truncated it. At least the php bit is working..
[maltfield@hetzner2 ~]$ sudo su -s /bin/sh apache -c "/usr/bin/php -v" PHP 5.6.33 (cli) (built: Jan 14 2018 08:07:11) Copyright (c) 1997-2016 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans [maltfield@hetzner2 ~]$
- slightly longer returns decent positive resultes too
[maltfield@hetzner2 ~]$ sudo su -s /bin/sh apache -c "/usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php" Cannot find config file [maltfield@hetzner2 ~]$
- with the config file it does nothing again
[maltfield@hetzner2 ~]$ sudo su -s /bin/sh apache -c "/usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php" [maltfield@hetzner2 ~]$
- I found better documentation on this cron, but it's not addressing my issue https://www.phplist.org/manual/ch036_setting-up-your-cron.xhtml
- Important to note: running this "-p processqueue" option over-and-over does not risk overloading the server; phplist will handle that
- I'll have to open a forum post about this..
Sun Feb 17, 2019
- I confirmed that our imported users into phplist don't have the "I agree to the OSE Privacy Policy" checkbox ticked
- I confirmed, most unfortunately, that after clicking the big green button titled "I agree to the new Privacy Policy / Yes keep sending me the OSEmail newsletter!" button in the repermission campaign, the "I agree to the OSE Privacy Policy" attribute for the subscriber does not change (it'll still be unticked for imported users)
- That said, the subscriber's History -> Subscription tab shows that they were added to the blacklist then removed from the blacklist "for manual confirmation of subscription". So we would probably have our ducks in a row here, since we could coorelate this specific event in the subscriber's history back to the campagin
- I asked about this 3 months ago, but it's gotten no responses https://discuss.phplist.org/t/re-permission-campaign-requiring-explicity-tos-action/4710
- In any case, new users will have the checkbox, and old users will either be blacklisted or necessarily have clicked our "I agree to the OSE Privacy Policy" button in order to be un-blacklisted
Fri Feb 15, 2019
- Marcin still wants better formatting for the transaction message = "Content of message subscribers receive when they subscribe"
- I added a colon to the end of the line "In order to provide you with this service we'll need to"
- Marcin didn't want the newlines to be inserted, but it's not in our config. They occur as part of a wordwrap after 75-80-ish characters. This is normal, and I think the negatives (ie: a paragraph of 200 words all appearing on one line) would outweigh the positives if we were to change this behavoiur)
- I finished my documentation Backblaze https://wiki.opensourceecology.org/wiki/Backblaze#Download_from_WUI
- I logged into the Backblaze B2 account and saw our 'ose-server-backups' bucket had 16 files at a total of 193G. Our bill last month was $0.78, and the upcoming bill is estimated at $0.86
Tue Feb 12, 2019
- Made some more changes to our phplist config per Marcin's requests
- I fixed a spelling mistake in the "State/Provence" attribute = 12. Changed to "State/Province"
- Deleted "Videography / Video Editing / Script Writing" option = 11 in attribte "Video Production / Video Editing / Script Writing / Explainer Videos " attribute = 16, as there was a redundant to the other option "Video Production / Video Editing / Script Writing / Explainer Videos" = 6.
- There are currently 33 skills is the current list of options for this attribute "Please identify which skills you possess"
Project Management Community Management Electrical Engineering Computer Aided Design (CAD; CAM; CAE) Computer Animation / Modeling Video Production / Video Editing / Script Writing / Explainer Videos Technical Writing / Documentation Graphics / Design / Infographics / Computer Graphics Fabrication / Digital Fabrication Machine Design Mechatronics Power Electronics Electrical Motor / Generator Design Electronics Hydraulic Motor Design Industrial Laser Design Industrial Robotics Design Solar Engineering Metallurgy Hot Metal Processing Tool & Die Precision Machine Design Wind Turbine Design Mechnical Engineering Agricultural Engineering Automotive Engineering Reliabilty Engineering Industrial Engineering Life Cycle Logistics Computer Programming / Database / CMS / Wiki Engineering Hydraulics / Pneumatics FreeCAD
- I changed attribute "City" = 11 to "City (so we can put you on a map)"
- Marcin wants the OSE new black/grey logo to appear at the top of the ose form here https://wiki.opensourceecology.org/wiki/File:OSE_Logo_-_Black.png
- he drew an example here https://docs.google.com/presentation/d/1gwaOiXfqc5jVx5-3VKVWDdEzjNqdahYPX5fkhy6TRYQ/edit#slide=id.g45ed589c64_0_0
- first, I downloaded the logo from the wiki link above and uploaded the logo to the wordpress site
- I got the wordpress-generated smaller versions of this icon:
-
- I checked the sizes of the above images. The thumbnail is unuseable as it's a square that cut off the left & right ends of the icon. The best option to use here is the 300x186 image.
- I made the logo appear at the top, shunk it to 215 pixels width, wrapped it in a 300 pixel div, and centered both the image & byline in the div.
<div style="width:300px; text-align:center;"> <img src="https://www.opensourceecology.org/wp-content/uploads/2012/05/OSE_Logo_-_Black-300x186.png" alt="Open Source Ecology Logo" width="215" class="aligncenter size-medium wp-image-10298" style="margin: 0 auto; display:block;" align="center"/> <strong>Open Source Blueprints for Civilization</strong> </div> <br />
- Marcin also wanted to customize the transaction message = "Content of message subscribers receive when they subscribe" as shown in this example https://wiki.opensourceecology.org/wiki/OSEmail#Subscribe
- I updated the subject from "Request for confirmation" to "Open Source Ecology Newsletters - Confirmation Required"
- Unfortunately, it's non-trivial to change the astrick-bulleted list to a numbered list. Perhaps that could be obtainable via a code change & PR, but I don't think that juice is worth the squeeze..
- Marcin also wanted to customize the transaction message = "Content of message subscribers receive when they subscribe" as shown in this example https://wiki.opensourceecology.org/wiki/OSEmail#Subscribe
You have been subscribed to the following newsletters: [LISTS] Please click the following link to confirm it's really you: [CONFIRMATIONURL] In order to provide you with this service we'll need to Transfer your contact information to phplist.opensourceecology.org Store your contact information in your phplist.opensourceecology.org account Send you emails from phplist.opensourceecology.org Track your interactions with these emails for marketing purposes If this is not correct, or you do not agree, simply take no action and delete this message.
- I tried changing this to the following, but it didn't work as expected. It appears that I can't use html in this message.
<h1>Open Source Ecology Newsletters - Confirmation Required</h1> Please confirm your email. You have been subscribed to the following newsletters: [LISTS] Please click the following link to confirm it's really you: [CONFIRMATIONURL] In order to provide you with this service we'll need to * Transfer your contact information to phplist.opensourceecology.org * Store your contact information in your phplist.opensourceecology.org account * Send you emails from phplist.opensourceecology.org * Track your interactions with these emails for marketing purposes If this is not correct, or you do not agree, simply take no action and delete this message.
- When I signed up, I got this back
Open Source Ecology Newsletters - Confirmation Required Please confirm your email. You have been subscribed to the following newsletters: * OSEmail Please click the following link to confirm it's really you: http://phplist.opensourceecology.org/lists/?p=confirm&uid=e018a9647ef71a498f4683adf5c10394 In order to provide you with this service we'll need to * Transfer your contact information to phplist.opensourceecology.org * Store your contact information in your phplist.opensourceecology.org account * Send you emails from phplist.opensourceecology.org * Track your interactions with these emails for marketing purposes If this is not correct, or you do not agree, simply take no action and delete this message.
- But I think that's sufficient. I'll send it to Marcin for approval.
Sat Feb 02, 2019
- Made some more changes to our phplist config per Marcin's requests
- I fixed a spelling mistake in the "State/Provence" attribute = 12. Changed to "State/Province"
- Deleted "Videography / Video Editing / Script Writing" option = 11 in attribte "Video Production / Video Editing / Script Writing / Explainer Videos " attribute = 16, as there was a redundant to the other option "Video Production / Video Editing / Script Writing / Explainer Videos" = 6.
- There are currently 33 skills is the current list of options for this attribute "Please identify which skills you possess"
Project Management Community Management Electrical Engineering Computer Aided Design (CAD; CAM; CAE) Computer Animation / Modeling Video Production / Video Editing / Script Writing / Explainer Videos Technical Writing / Documentation Graphics / Design / Infographics / Computer Graphics Fabrication / Digital Fabrication Machine Design Mechatronics Power Electronics Electrical Motor / Generator Design Electronics Hydraulic Motor Design Industrial Laser Design Industrial Robotics Design Solar Engineering Metallurgy Hot Metal Processing Tool & Die Precision Machine Design Wind Turbine Design Mechnical Engineering Agricultural Engineering Automotive Engineering Reliabilty Engineering Industrial Engineering Life Cycle Logistics Computer Programming / Database / CMS / Wiki Engineering Hydraulics / Pneumatics FreeCAD
- I changed attribute "City" = 11 to "City (so we can put you on a map)"
- Marcin wants the OSE new black/grey logo to appear at the top of the ose form here https://wiki.opensourceecology.org/wiki/File:OSE_Logo_-_Black.png
- he drew an example here https://docs.google.com/presentation/d/1gwaOiXfqc5jVx5-3VKVWDdEzjNqdahYPX5fkhy6TRYQ/edit#slide=id.g45ed589c64_0_0
- first, I downloaded the logo from the wiki link above and uploaded the logo to the wordpress site
- I got the wordpress-generated smaller versions of this icon:
-
- I checked the sizes of the above images. The thumbnail is unuseable as it's a square that cut off the left & right ends of the icon. The best option to use here is the 300x186 image.
- I made the logo appear at the top, shunk it to 215 pixels width, wrapped it in a 300 pixel div, and centered both the image & byline in the div.
<div style="width:300px; text-align:center;"> <img src="https://www.opensourceecology.org/wp-content/uploads/2012/05/OSE_Logo_-_Black-300x186.png" alt="Open Source Ecology Logo" width="215" class="aligncenter size-medium wp-image-10298" style="margin: 0 auto; display:block;" align="center"/> <strong>Open Source Blueprints for Civilization</strong> </div> <br />
- Marcin also wanted to customize the transaction message = "Content of message subscribers receive when they subscribe" as shown in this example https://wiki.opensourceecology.org/wiki/OSEmail#Subscribe
- I updated the subject from "Request for confirmation" to "Open Source Ecology Newsletters - Confirmation Required"
- Unfortunately, it's non-trivial to change the astrick-bulleted list to a numbered list. Perhaps that could be obtainable via a code change & PR, but I don't think that juice is worth the squeeze..
- Marcin also wanted to customize the transaction message = "Content of message subscribers receive when they subscribe" as shown in this example https://wiki.opensourceecology.org/wiki/OSEmail#Subscribe
You have been subscribed to the following newsletters: [LISTS] Please click the following link to confirm it's really you: [CONFIRMATIONURL] In order to provide you with this service we'll need to Transfer your contact information to phplist.opensourceecology.org Store your contact information in your phplist.opensourceecology.org account Send you emails from phplist.opensourceecology.org Track your interactions with these emails for marketing purposes If this is not correct, or you do not agree, simply take no action and delete this message.
- I tried changing this to the following, but it didn't work as expected. It appears that I can't use html in this message.
<h1>Open Source Ecology Newsletters - Confirmation Required</h1> Please confirm your email. You have been subscribed to the following newsletters: [LISTS] Please click the following link to confirm it's really you: [CONFIRMATIONURL] In order to provide you with this service we'll need to * Transfer your contact information to phplist.opensourceecology.org * Store your contact information in your phplist.opensourceecology.org account * Send you emails from phplist.opensourceecology.org * Track your interactions with these emails for marketing purposes If this is not correct, or you do not agree, simply take no action and delete this message.
- When I signed up, I got this back
Open Source Ecology Newsletters - Confirmation Required Please confirm your email. You have been subscribed to the following newsletters: * OSEmail Please click the following link to confirm it's really you: http://phplist.opensourceecology.org/lists/?p=confirm&uid=e018a9647ef71a498f4683adf5c10394 In order to provide you with this service we'll need to * Transfer your contact information to phplist.opensourceecology.org * Store your contact information in your phplist.opensourceecology.org account * Send you emails from phplist.opensourceecology.org * Track your interactions with these emails for marketing purposes If this is not correct, or you do not agree, simply take no action and delete this message.
- But I think that's sufficient. I'll send it to Marcin for approval.
Tue Feb 05, 2019
- Marcin responded to my email about the updated wordpress post about phplist
- he asked if I could make the link to the privacy policy open it in a new window, which I totally agree with. I did this by changing the attribute #4 = "I agree to the OSE Privacy Policy." anchor link to include the attribute "target" with value "_blank" in the phplist wui. I also changed it in the html within the wp post
I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy' target='_blank'>Privacy Policy</a>.
- Marcin asked me to change the response after signup to include a period in the sentence "Thanks, you have been added to our newsletter."
- he also noted that the javascript includes a variable named "successMessage" with a distinct message. Indeed, the text that's displayed comes as a response to the ajax query to the phplist server, and this response is configurable from the phplist wui at Config -> Settings -> "Text to display when subscription with an AJAX request was successful". I changed it to:
- Marcin asked me to change the response after signup to include a period in the sentence "Thanks, you have been added to our newsletter."
<h3>Thanks, you have been added to our newsletter.</h3><p>You will receive an email to confirm your subscription. Please click the link in the email to confirm.</p>
- that actually didn't work. it looks like that's a default setting, but there's also per-subscribe-list settings for this response. I changed this via Config -> Subscribe Pages -> Edit (for id=3, which matches the URL used in the html of the post) -> "Text to display when subscription with an AJAX request was successful". I changed this to:
<h3>Thanks, you have been added to our newsletter.</h3><p>You will receive an email to confirm your subscription. Please click the link in the email to confirm.</p>
- Marcin provided a description of the Design Sprints newsletter as well, which I added to the subscribe list's config in phplist
- I updated the wiki documentation's example of an ajax form with the changes we made https://wiki.opensourceecology.org/wiki/Phplist#Example_Form_for_OSEmail
- ...
- The phplist team asked me to cherry pick a git commit for a PR I submitted, but I couldn't get it to work; I followed-up https://github.com/phpList/phplist3/pull/445#issuecomment-460923780
Sat Jan 26, 2019
- I logged into backblaze b2 to ensure that my nightly backups from hetnzer2 are still coming in after I've deprecated the dreamhost backup script & replaced it with the backblaze one
- I see that there's a file named "daily_hetzner2_20190126_072001.tar.gpg (started large file)" which I guess is what it looks like when a backup is _currently_ being uploaded.
- I also see that there's a file from the 25th, yesterday--and the days leading up to it. My change was last week, so this looks good.
- The whole 'ose-server-backups' bucket is 190.5 GB with 16 files
- The billing section shows that we paid our $0.78 fee for the first month, and that it's estimating a $0.28 fee for the second month. pretty fucking good.
- I also confirmed that the backup log file (/var/log/backups/backup.log) contains my new comments, which makes the `time` outputs much more useful. And the --noProgress argument has worked wonderfully--the log file is now showing a size of 4kb in du.
- ...
- Marcin drafted a new post to our main ose wordpress site about or transition to phplist. We want to publish this before issuing the repermission campaign.
- I edited the post to add the ajax form in the post, so that new users just learning about the OSEmail newsletter could signup with phplist
- I hit some modsecurity false-positives
- 958052 xss
- 958407 xss
- 973305 xss
- 973307 xss
- 973307 xss
- even though I'm editing the post in the "text" tab (not "visual"), wordpress is still adding invisible "
" tags into thet post. For example, it does it in the middle of a javascript function, which creates syntax errors and compeletly breaks the ajax form
- I went to my user profile & checked the box "DIsable the visual editorwhen editing"
- I removed my ajax form block. saved. added it again. saved. It's still injecting these paragraph tags!
- I hit some modsecurity false-positives
...
</noscript></p>
<p><script type="text/javascript">
function checkform() {</p>
<p> // first, clear the response div from the previous attempts results
jQuery("#result").empty();</p>
...
- I found some documentation on this behaviour here https://codex.wordpress.org/Function_Reference/wpautop#Disabling_the_filter
- We probably don't want to wholesale disable it for the entire blog, so the best option is probably to use this simple plugin https://wordpress.org/plugins/wpautop-control/
- I installed & activated this plugin with wp-cli for the osemain
[maltfield@hetzner2 htdocs]$ sudo -u wp -i wp --path=/var/www/html/www.opensourceecology.org/htdocs plugin install --activate wpautop-control ...
- # To enable it, I had to go to edit the post and enable the "Custom Fields" checkbox under "Screen Options" at the top. Then I scrolled down to the "Custom Fields" -> "Add New Custom Field:" section, clicked "Enter new", and typed "wpautop" for the Name and "false" for the Value.
- When I reloaded the post's preview, the paragraphs were all running together. Success! I manually added the necessary <p> tags around Marcin's paragraphs. I saved, refreshed the preview, and confirmed that it looks good
- I replaced the REGISTRATION LINK text with our ajax form code block, saved, refreshed the preview, and now the form appears!
- unfortunately, the form is broken. clicking the "SUBSCRIBE" button (which, interestingly, looks different as it's inhereted the css from our osemain website) produces an error that the checkform() function is not defined
ReferenceError: checkform is not defined[Learn More]
- it appears to be unhappy about a syntax error in the middle of a comment. It looks like my multi-line comment tag is the html syntax, but I should really use the JS syntax. I changed it
- now it works better. I confirmed when I don't click the box "I agree to the OSE Privcacy Policy" that it pops-up with the error as expected. But when I do check the box, I get a pop-up "Sorry, we were unable to process your subscription." and the JS console shows that it's because of a Cross-Origin Request being blocked. Right, I need to add both www.opensourceecology.org _and_ microfactory.opensourceecology.org into the whitelist of phplist.opensourceecology.org's headers in the phplist config file
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=3. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘https://microfactory.opensourceecology.org’).
- well shit, it looks like the fucking Access-Control-Allow-Origin header only supports a single domain! The w3c recommends dynamically responding with the single approved domain using logic to determine who has requested the content and if they're in the whitelist. That sucks! https://www.w3.org/TR/cors/#resource-implementation
- fortunately, we can do this in nginx--which would occur before the varnish cache https://stackoverflow.com/questions/1653308/access-control-allow-origin-multiple-origin-domains/12414239#12414239
- I commented-out my line in the phplist config.php setting ACCESS_CONTROL_ALLOW_ORIGIN. This reset it back to "http://phplist.opensourceecology.org"
- I spent much time trying to get the nginx config to set the ACCESS_CONTROL_ALLOW_ORIGN header, but it never changed from "http://phplist.opensourceecology.org". Indeed, it appears that nginx won't set headers when using proxy_pass https://stackoverflow.com/questions/14501047/how-to-add-a-response-header-on-nginx-when-using-proxy-pass
- hmm, actually, for some reason the nginx config file that's applying for phplist is 'awstats.opensourceecology.org'. No wonder no changes I make are applying..
- doh! the filename was wong; it doesn't end in .conf!
root@hetzner2 conf.d]# date Sat Jan 26 12:26:26 UTC 2019 [root@hetzner2 conf.d]# pwd /etc/nginx/conf.d [root@hetzner2 conf.d]# ls -1 phplist* phplist.opensourceecology.org phplist.opensourceecology.org.4443.disabled [root@hetzner2 conf.d]# ls -1 awstats* awstats.openbuildinginstitute.org.conf awstats.opensourceecology.org.conf [root@hetzner2 conf.d]#
- I fixed this with the following nginx config
[root@hetzner2 ~]# grep -A14 'location /' /etc/nginx/conf.d/phplist.opensourceecology.org.conf
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
# handle cors whitelist for ajax subscription to phplist
proxy_hide_header Access-Control-Allow-Origin;
if ( $http_origin ~ "^https://(www.opensourceecology.org|microfactory.opensourceecology.org)$" ) {
add_header Access-Control-Allow-Origin $http_origin;
}
}
- I also updated the phplist config with a comment pointing out that this logic is now in nginx, not php
[root@hetzner2 ~]# grep -B8 "define('ACCESS_CONTROL_ALLOW_ORIGIN" /var/www/html/phplist.opensourceecology.org/config.php
// allow AJAX queries to add subscribers to our db from other domains
// Note: The ACCESS_CONTROL_ALLOW_ORIGIN header does not support multiple
// domains, so we instead have to maintain a whitelist logically and
// dynamically return the relevant domain iff it's in the whitelist.
// Therefore, we actually override this phplist ACCESS_CONTROL_ALLOW_ORIGIN
// header in our nginx config. See the relevant nginx config file:
// * /etc/nginx/conf.d/phplist.opensourceecology.org.conf
#define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://www.opensourceecology.org" );
[root@hetzner2 ~]#
- It works! I made some further changes to it so that it looks good in the osemain wordpress post, and finally documented the final ajax form block on the wiki https://wiki.opensourceecology.org/wiki/Phplist#Ajax_Form
- I also documented the CORS stuff above https://wiki.opensourceecology.org/wiki/Phplist#Cross-Origin_Resource_Sharing
- I sent Marcin an email asking for review of the updated post & for a description of the Design Sprints list
- I updated the description of the OSEmal list with what Marcin provided in the blog post
OSEmail is our main OSE Newsletter featuring news updates, workshop announcement, progress reports, and other noteworthy items. OSEmail comes out a few times per year at monthly or longer intervals. Anyone can sign up to receive our free newsletter. You can see more information at https://wiki.opensourceecology.org/wiki/OSEmail
- This is now reflected in our main subscription signup page on phplist's site (the one that we'll link to for people who want to subscribe to the Design Sprints list) https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
Wed Jan 16, 2019
- I logged into the server and found that the backup was currently uploading today's nightly backup to backblaze b2
- The logs are being hella spammed with the upload progress bar (the backup log containing just today's backup was already 14M), so I updated the backup.sh script to use the --noProgress argument.
[root@hetzner2 ~]# du -sh /var/log/backups/* 6.2M /var/log/backups/backup.log 48K /var/log/backups/backup.log-20181206.gz 96K /var/log/backups/backup.log-20181207.gz 48K /var/log/backups/backup.log-20181209.gz 48K /var/log/backups/backup.log-20181210.gz 96K /var/log/backups/backup.log-20181211.gz 348K /var/log/backups/backup.log-20181213.gz 1012K /var/log/backups/backup.log-20181215.gz 340K /var/log/backups/backup.log-20181216.gz 356K /var/log/backups/backup.log-20181218.gz 748K /var/log/backups/backup.log-20181219.gz 364K /var/log/backups/backup.log-20181221.gz 372K /var/log/backups/backup.log-20181222.gz 716K /var/log/backups/backup.log-20181223.gz 712K /var/log/backups/backup.log-20181225.gz 352K /var/log/backups/backup.log-20181226.gz 392K /var/log/backups/backup.log-20181227.gz 352K /var/log/backups/backup.log-20181228.gz 352K /var/log/backups/backup.log-20181230.gz 356K /var/log/backups/backup.log-20181231.gz 360K /var/log/backups/backup.log-20190101.gz 360K /var/log/backups/backup.log-20190102.gz 700K /var/log/backups/backup.log-20190103.gz 352K /var/log/backups/backup.log-20190105.gz 352K /var/log/backups/backup.log-20190106.gz 364K /var/log/backups/backup.log-20190107.gz 388K /var/log/backups/backup.log-20190108.gz 408K /var/log/backups/backup.log-20190109.gz 360K /var/log/backups/backup.log-20190110.gz 488K /var/log/backups/backup.log-20190111.gz 352K /var/log/backups/backup.log-20190113.gz 384K /var/log/backups/backup.log-20190114.gz 14M /var/log/backups/backup.log-20190115 [root@hetzner2 ~]# 79M . [root@hetzner2 ~]#
- I also added many INFO echos to the script to make the logs more useful
- I ssh'd into our dreamhost server. Our ose_marcin user's home directly is currently using 304G of disk space. Hetzner1 backups are at 13G & hetzner2 backups are at 288G
hancock% pwd /home/marcin_ose hancock% date Wed Jan 16 02:04:22 PST 2019 hancock% du -sh . 304G . hancock% du -sh hetzner1 13G hetzner1 hancock% du -sh hetzner2 288G hetzner2 hancock%
- I confirmed that the backup from today (20190116) was not present on the dreamhost server. success!
hancock% du -sh hetzner1/* 12G hetzner1/20180501-052002 259M hetzner1/20180502-052001 464M hetzner1/20180602-052001 464M hetzner1/20180702-052001 hancock% du -sh hetzner2/* 15G hetzner2/20180501-072001 15G hetzner2/20180601_072001 15G hetzner2/20180701_072001 16G hetzner2/20180801_072001 17G hetzner2/20180901_072001 17G hetzner2/20181001_072001 17G hetzner2/20181101_072001 17G hetzner2/20181202_072001 33G hetzner2/20190101_072001 33G hetzner2/20190112_072001 33G hetzner2/20190113_072001 33G hetzner2/20190114_072001 33G hetzner2/20190115_072001 hancock%
Tue Jan 15, 2019
- I renamed the /root/backups/backup.sh script to be 'backup.old.20180115.sh' and I renamed backup2.sh to be 'backup.sh'
- I also renamed /etc/cron.d/backup_to_dreamhost to be 'backup_to_backblaze'
- The above has the effect of disabling backups being sent to dreamhost. All backups will now only go to backblaze b2 going forward.
Thr Jan 10, 2019
- checked backblaze wui
- total size of bucket is currently 157.7G
- I confirmed that th yearly backup file exists at 17.5G = yearly_hetzner2_20190101_111520.tar.gpg
- I checked our billing; in all of 2018 it was $0.02. In 2019, it's $0.63 so far in 2019. (which is actually the billing period between Dec 16 through Jan 10).
- I should do a test downloading the 2019-01-01 file and document the process (with screenshots) on our wiki after 2019-01-17--since we get a free quota of downloads once per month, and I used some of it for testing/documenting the download process via the cli in this billing period (which cost $0.61 to do)
Wed Jan 09, 2019
- marked time
- still waiting on Marcin to unblock me for phplist
