Maltfield Log/2018 Q4
My work log from the year 2018 Quarter 4. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
Contents
- 1 See Also
- 2 Thr Dec 24, 2018
- 3 Wed Dec 23, 2018
- 4 Wed Dec 23, 2018
- 5 Tue Dec 18, 2018
- 6 Mon Dec 17, 2018
- 7 Sun Dec 16, 2018
- 8 Sat Dec 15, 2018
- 9 Fri Dec 14, 2018
- 10 Wed Dec 12, 2018
- 11 Mon Dec 10, 2018
- 12 Sun Dec 09, 2018
- 13 Thr Dec 06, 2018
- 14 Wed Dec 05, 2018
- 15 Mon Dec 03, 2018
- 16 Tue Nov 29, 2018
- 17 Wed Nov 28, 2018
- 18 Tue Nov 27, 2018
- 19 Mon Nov 26, 2018
- 20 Sat Nov 24, 2018
- 21 Fri Nov 23, 2018
- 22 Thr Nov 22, 2018
- 23 Sun Nov 07, 2018
- 24 Sun Nov 04, 2018
- 25 Fri Nov 02, 2018
- 26 Tue Oct 30, 2018
- 27 Mon Oct 29, 2018
- 28 Fri Oct 26, 2018
- 29 Wed Oct 24, 2018
- 30 Tue Oct 23, 2018
- 31 Mon Oct 22, 2018
- 32 Sat Oct 20, 2018
- 33 Fri Oct 19, 2018
- 34 Sun Oct 14, 2018
- 35 Mon Oct 01, 2018
See Also
Thr Dec 24, 2018
- documented how to install the b2 cli https://wiki.opensourceecology.org/wiki/Backblaze#Install_CLI
sudo su - # we must install the b2 command in a virtualenv from the binary. if we do it via easy_install, it will # break other critical python packages, such as certbot (yielding our websites down when the https certs # fail to auto-renew properly) # for more info: https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q3#Wed_Aug_08.2C_2018 yum install python-virtualenv # and install other depends yum install python-setuptools adduser b2user sudo su - b2user mkdir virtualenv cd virtualenv/ virtualenv . cd .. mkdir sandbox git clone https://github.com/Backblaze/B2_Command_Line_Tool.git cd B2_Command_Line_Tool/ python setup.py install # test it ~/virtualenv/bin/b2 version
Wed Dec 23, 2018
- I found 4 of the Design Sprint Survey responders in the spreadsheet had used 2x email addresses (csv) in the email address field, which was directly imported into phplist (broken). I found these and deleted the non-gmail entry for both
- I encountered a modsecurity false-positive
- whitelisted 950120, generic attack
- Marcin hasn't responded to my request asking for a breif, public description of each of the OSEmail & Design Sprints lists
- I'll continue working on importing the other tabs from the Design Sprints spreadsheet into phplist
- I created a new attribute (type = textarea) so we can make internal notes about subscribers, in lieu of a proper CRM tool. It's attribute #22 = Internal Notes
- The only other tab on this Design Sprint spreadsheet that has unique data are the "Sheet 2" sheet with 123 rows. The "Team Ironworker" tab is empty, and both the "Mailchimp Import" and "Team Backhoe" tabs' contents literally only references the first tab's data (Survey Results).
- I imported the data into phplist. looks like 7 subscribers were added
7 emails succesfully imported to the database and added to 1 lists. 7 emails subscribed to the lists 116 emails already existed in the database Subscriber data was updated for 123 subscribers 0 subscribers were matched by foreign key, 123 by email
- spent some time adjusting the order of the attributes displayed on the newsletter subscribe page so they make sense https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
- Also gave the header & footer the OSE branding as was used in the OSEmail signup page
- I sent an email to Marcin asking him for a drafted wordpress blog post about the phplist roll-out & 2x descriptions of each of the OSEmail & Design Sprints lists
- ...
- I checked the backblaze b2 account via the wui, and I see that we have 3x dailys at the new size of 17.5G and one weekly at this size.
- The total size of the bucket is now 105.1G with 14 files.
- The Billing section shows $0.12 in estimated charges
- I spent some time updating the wiki documentation of our backups, including testing a download & decryption test of a full backup from backblaze b2 to hetnzer2
[maltfield@hetzner2 backblaze]$ sudo su - b2user [sudo] password for maltfield: Last login: Sat Nov 24 13:10:42 UTC 2018 on pts/124 [b2user@hetzner2 ~]$ ~/virtualenv/bin/b2 ls ose-server-backups daily_hetzner2_20181221_111534.tar.gpg daily_hetzner2_20181222_111516.tar.gpg daily_hetzner2_20181223_111501.tar.gpg monthly_hetzner2_20181001_091809.tar.gpg monthly_hetzner2_20181101_091810.tar.gpg monthly_hetzner2_20181201_091759.tar.gpg weekly_hetzner2_20181126_091805.tar.gpg weekly_hetzner2_20181203_091823.tar.gpg weekly_hetzner2_20181210_091822.tar.gpg weekly_hetzner2_20181217_111545.tar.gpg [b2user@hetzner2 tmp]$ tmpDir="/var/tmp/backblazeRestore_`date +%Y%m%d_%H%M%S`" [b2user@hetzner2 tmp]$ cd $tmpDir [b2user@hetzner2 backblazeRestore_20181223_135712]$ ~/virtualenv/bin/b2 download-file-by-name ose-server-backups daily_hetzner2_20181223_111501.tar.gpg daily_hetzner2_20181223_111501.tar.gpg daily_hetzner2_20181223_111501.tar.gpg: 100%|| 17.5G/17.5G [19:29<00:00, 15.0MB/s] File name: daily_hetzner2_20181223_111501.tar.gpg File id: 4_z5605817c251dadb96e4d0118_f206daef4188682f6_d20181223_m113926_c001_v0001106_t0057 File size: 17509827199 Content type: application/octet-stream Content sha1: none INFO src_last_modified_millis: 1545565162536 [b2user@hetzner2 backblazeRestore_20181223_135712]$ du -sh * 17G daily_hetzner2_20181223_111501.tar.gpg ... [maltfield@hetzner2 backblaze]$ sudo su - [sudo] password for maltfield: Last login: Sun Dec 23 12:09:45 UTC 2018 on pts/113 [root@hetzner2 ~]# cd /var/tmp/back backblaze/ backblazeRestore_20181223_135712/ backups_for_migration_from_hetzner1/ [root@hetzner2 ~]# cd /var/tmp/backblazeRestore_20181223_135712/ [root@hetzner2 backblazeRestore_20181223_135712]# du -sh * 17G daily_hetzner2_20181223_111501.tar.gpg [root@hetzner2 backblazeRestore_20181223_135712]# gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --output daily_hetzner2_20181223_111501.tar daily_hetzner2_20181223_111501.tar.gpg gpg: AES256 encrypted data gpg: encrypted with 1 passphrase [root@hetzner2 backblazeRestore_20181223_135712]# du -sh * 17G daily_hetzner2_20181223_111501.tar 17G daily_hetzner2_20181223_111501.tar.gpg [root@hetzner2 backblazeRestore_20181223_135712]# tar -xvf daily_hetzner2_20181223_111501.tar root/backups/sync/daily_hetzner2_20181223_111501/etc/ root/backups/sync/daily_hetzner2_20181223_111501/etc/etc.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/home/ root/backups/sync/daily_hetzner2_20181223_111501/home/home.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/log/ root/backups/sync/daily_hetzner2_20181223_111501/log/log.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/mysqldump/ root/backups/sync/daily_hetzner2_20181223_111501/mysqldump/mysqldump.20181223_111501.sql.gz root/backups/sync/daily_hetzner2_20181223_111501/root/ root/backups/sync/daily_hetzner2_20181223_111501/root/root.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/www/ root/backups/sync/daily_hetzner2_20181223_111501/www/www.20181223_111501.tar.gz [root@hetzner2 backblazeRestore_20181223_135712]# find root/backups/sync/daily_hetzner2_20181223_111501/ -type f | xargs du -sh 12M root/backups/sync/daily_hetzner2_20181223_111501/etc/etc.20181223_111501.tar.gz 2.1M root/backups/sync/daily_hetzner2_20181223_111501/root/root.20181223_111501.tar.gz 110M root/backups/sync/daily_hetzner2_20181223_111501/log/log.20181223_111501.tar.gz 517M root/backups/sync/daily_hetzner2_20181223_111501/mysqldump/mysqldump.20181223_111501.sql.gz 16G root/backups/sync/daily_hetzner2_20181223_111501/www/www.20181223_111501.tar.gz 108M root/backups/sync/daily_hetzner2_20181223_111501/home/home.20181223_111501.tar.gz [root@hetzner2 backblazeRestore_20181223_135712]#
- I checked the billing section again, and now I see a $0.17 charge for "Download Bandwidth". Note that Backblaze B2 gives 10G of free downloads per month.
Wed Dec 23, 2018
- I found 4 of the Design Sprint Survey responders in the spreadsheet had used 2x email addresses (csv) in the email address field, which was directly imported into phplist (broken). I found these and deleted the non-gmail entry for both
- I encountered a modsecurity false-positive
- whitelisted 950120, generic attack
- Marcin hasn't responded to my request asking for a breif, public description of each of the OSEmail & Design Sprints lists
- I'll continue working on importing the other tabs from the Design Sprints spreadsheet into phplist
- I created a new attribute (type = textarea) so we can make internal notes about subscribers, in lieu of a proper CRM tool. It's attribute #22 = Internal Notes
- The only other tab on this Design Sprint spreadsheet that has unique data are the "Sheet 2" sheet with 123 rows. The "Team Ironworker" tab is empty, and both the "Mailchimp Import" and "Team Backhoe" tabs' contents literally only references the first tab's data (Survey Results).
- I imported the data into phplist. looks like 7 subscribers were added
7 emails succesfully imported to the database and added to 1 lists. 7 emails subscribed to the lists 116 emails already existed in the database Subscriber data was updated for 123 subscribers 0 subscribers were matched by foreign key, 123 by email
- spent some time adjusting the order of the attributes displayed on the newsletter subscribe page so they make sense https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
- Also gave the header & footer the OSE branding as was used in the OSEmail signup page
- I sent an email to Marcin asking him for a drafted wordpress blog post about the phplist roll-out & 2x descriptions of each of the OSEmail & Design Sprints lists
- ...
- I checked the backblaze b2 account via the wui, and I see that we have 3x dailys at the new size of 17.5G and one weekly at this size.
- The total size of the bucket is now 105.1G with 14 files.
- The Billing section shows $0.12 in estimated charges
- I spent some time updating the wiki documentation of our backups, including testing a download & decryption test of a full backup from backblaze b2 to hetnzer2
[maltfield@hetzner2 backblaze]$ sudo su - b2user [sudo] password for maltfield: Last login: Sat Nov 24 13:10:42 UTC 2018 on pts/124 [b2user@hetzner2 ~]$ ~/virtualenv/bin/b2 ls ose-server-backups daily_hetzner2_20181221_111534.tar.gpg daily_hetzner2_20181222_111516.tar.gpg daily_hetzner2_20181223_111501.tar.gpg monthly_hetzner2_20181001_091809.tar.gpg monthly_hetzner2_20181101_091810.tar.gpg monthly_hetzner2_20181201_091759.tar.gpg weekly_hetzner2_20181126_091805.tar.gpg weekly_hetzner2_20181203_091823.tar.gpg weekly_hetzner2_20181210_091822.tar.gpg weekly_hetzner2_20181217_111545.tar.gpg [b2user@hetzner2 tmp]$ tmpDir="/var/tmp/backblazeRestore_`date +%Y%m%d_%H%M%S`" [b2user@hetzner2 tmp]$ cd $tmpDir [b2user@hetzner2 backblazeRestore_20181223_135712]$ ~/virtualenv/bin/b2 download-file-by-name ose-server-backups daily_hetzner2_20181223_111501.tar.gpg daily_hetzner2_20181223_111501.tar.gpg daily_hetzner2_20181223_111501.tar.gpg: 100%|| 17.5G/17.5G [19:29<00:00, 15.0MB/s] File name: daily_hetzner2_20181223_111501.tar.gpg File id: 4_z5605817c251dadb96e4d0118_f206daef4188682f6_d20181223_m113926_c001_v0001106_t0057 File size: 17509827199 Content type: application/octet-stream Content sha1: none INFO src_last_modified_millis: 1545565162536 [b2user@hetzner2 backblazeRestore_20181223_135712]$ du -sh * 17G daily_hetzner2_20181223_111501.tar.gpg ... [maltfield@hetzner2 backblaze]$ sudo su - [sudo] password for maltfield: Last login: Sun Dec 23 12:09:45 UTC 2018 on pts/113 [root@hetzner2 ~]# cd /var/tmp/back backblaze/ backblazeRestore_20181223_135712/ backups_for_migration_from_hetzner1/ [root@hetzner2 ~]# cd /var/tmp/backblazeRestore_20181223_135712/ [root@hetzner2 backblazeRestore_20181223_135712]# du -sh * 17G daily_hetzner2_20181223_111501.tar.gpg [root@hetzner2 backblazeRestore_20181223_135712]# gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --output daily_hetzner2_20181223_111501.tar daily_hetzner2_20181223_111501.tar.gpg gpg: AES256 encrypted data gpg: encrypted with 1 passphrase [root@hetzner2 backblazeRestore_20181223_135712]# du -sh * 17G daily_hetzner2_20181223_111501.tar 17G daily_hetzner2_20181223_111501.tar.gpg [root@hetzner2 backblazeRestore_20181223_135712]# tar -xvf daily_hetzner2_20181223_111501.tar root/backups/sync/daily_hetzner2_20181223_111501/etc/ root/backups/sync/daily_hetzner2_20181223_111501/etc/etc.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/home/ root/backups/sync/daily_hetzner2_20181223_111501/home/home.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/log/ root/backups/sync/daily_hetzner2_20181223_111501/log/log.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/mysqldump/ root/backups/sync/daily_hetzner2_20181223_111501/mysqldump/mysqldump.20181223_111501.sql.gz root/backups/sync/daily_hetzner2_20181223_111501/root/ root/backups/sync/daily_hetzner2_20181223_111501/root/root.20181223_111501.tar.gz root/backups/sync/daily_hetzner2_20181223_111501/www/ root/backups/sync/daily_hetzner2_20181223_111501/www/www.20181223_111501.tar.gz [root@hetzner2 backblazeRestore_20181223_135712]# find root/backups/sync/daily_hetzner2_20181223_111501/ -type f | xargs du -sh 12M root/backups/sync/daily_hetzner2_20181223_111501/etc/etc.20181223_111501.tar.gz 2.1M root/backups/sync/daily_hetzner2_20181223_111501/root/root.20181223_111501.tar.gz 110M root/backups/sync/daily_hetzner2_20181223_111501/log/log.20181223_111501.tar.gz 517M root/backups/sync/daily_hetzner2_20181223_111501/mysqldump/mysqldump.20181223_111501.sql.gz 16G root/backups/sync/daily_hetzner2_20181223_111501/www/www.20181223_111501.tar.gz 108M root/backups/sync/daily_hetzner2_20181223_111501/home/home.20181223_111501.tar.gz [root@hetzner2 backblazeRestore_20181223_135712]#
- I checked the billing section again, and now I see a $0.17 charge for "Download Bandwidth". Note that Backblaze B2 gives 10G of free downloads per month.
Tue Dec 18, 2018
- Marcin confirmed that the latest test of the phplist repermission campaign (with base64 + attached images) displayed correctly in hist gmail. great!
- Marcin also confirmed that "Novice" should be considered a check for the skill
- ...
- Continued my awk hacking to convert this spreadsheet into something importable into the new data types used in phplist
- So far the POC is a bit more complex, so I put it in a script
user@ose:~/tmp/phplist/listImport/designSprints$ cat csvToPhplist.sh
#!/bin/bash
cat designSprints.1.csv | awk -F "\t" 'BEGIN {ORS=","} { \
print $1 "\t" $2 "\t" $3 "\t" $4 "\t" $5 "\t" $6 "\t" $7 "\t" $8 "\t" $9 "\t"
if ($10 == "Intermediate" || $10 == "Advanced" || $10 == "Novice"){ print "Project Management" } else {print ""} \
if ($11 == "Intermediate" || $11 == "Advanced" || $11 == "Novice"){ print "Computer Aided Design" } else {print ""} \
print "\n"
}' | \
sed 's/,,/,/g' | \
sed 's/\t,/\t/g' | \
sed 's/,\t/\t/g'
exit 0
user@ose:~/tmp/phplist/listImport/designSprints$
- ugh, this spreadsheet has a bunch of columns with width=0, which makes matching the encoding painful.
- a bit more hacking gets all the columns lined up
user@ose:~/tmp/phplist/listImport/designSprints$ cat csvToPhplist.sh
#!/bin/bash
cat designSprints.1.csv | awk -F "\t" 'BEGIN {ORS=","} { \
print $1 "\t" $2 "\t" $3 "\t" $4 "\t" $5 "\t" $6 "\t" $7 "\t" $8 "\t" $9 "\t"
if ($10 == "Intermediate" || $10 == "Advanced" || $10 == "Novice"){ print "Project Management" } else {print ""} \
if ($11 == "Intermediate" || $11 == "Advanced" || $11 == "Novice"){ print "Community Management" } else {print ""} \
if ($12 == "Intermediate" || $12 == "Advanced" || $12 == "Novice"){ print "Electrical Engineering" } else {print ""} \
if ($13 == "Intermediate" || $13 == "Advanced" || $13 == "Novice"){ print "CAD" } else {print ""} \
if ($14 == "Intermediate" || $14 == "Advanced" || $14 == "Novice"){ print "Computer Animation" } else {print ""} \
if ($15 == "Intermediate" || $15 == "Advanced" || $15 == "Novice"){ print "Video Production" } else {print ""} \
if ($16 == "Intermediate" || $16 == "Advanced" || $16 == "Novice"){ print "Technical Writing" } else {print ""} \
if ($17 == "Intermediate" || $17 == "Advanced" || $17 == "Novice"){ print "Graphics" } else {print ""} \
if ($18 == "Intermediate" || $18 == "Advanced" || $18 == "Novice"){ print "Fabrication" } else {print ""} \
# we skip a few empty columns
if ($23 == "Intermediate" || $23 == "Advanced" || $23 == "Novice"){ print "Machine Design" } else {print ""} \
if ($24 == "Intermediate" || $24 == "Advanced" || $24 == "Novice"){ print "Mechatronics" } else {print ""} \
if ($25 == "Intermediate" || $25 == "Advanced" || $25 == "Novice"){ print "Power Electronics" } else {print ""} \
if ($26 == "Intermediate" || $26 == "Advanced" || $26 == "Novice"){ print "Electrical Motor Design" } else {print ""} \
if ($27 == "Intermediate" || $27 == "Advanced" || $27 == "Novice"){ print "Electronics" } else {print ""} \
if ($28 == "Intermediate" || $28 == "Advanced" || $28 == "Novice"){ print "Hydraulic Motor Design" } else {print ""} \
# we skip the duplicate & empty column for 29
if ($30 == "Intermediate" || $30 == "Advanced" || $30 == "Novice"){ print "Industrial Lasar Design" } else {print ""} \
if ($31 == "Intermediate" || $31 == "Advanced" || $31 == "Novice"){ print "Industrail Robotics Design" } else {print ""} \
if ($32 == "Intermediate" || $32 == "Advanced" || $32 == "Novice"){ print "Solar Engineering" } else {print ""} \
if ($33 == "Intermediate" || $33 == "Advanced" || $33 == "Novice"){ print "Metallurgy" } else {print ""} \
if ($34 == "Intermediate" || $34 == "Advanced" || $34 == "Novice"){ print "Hot Metal Processing" } else {print ""} \
if ($35 == "Intermediate" || $35 == "Advanced" || $35 == "Novice"){ print "Tool and Die" } else {print ""} \
if ($36 == "Intermediate" || $36 == "Advanced" || $36 == "Novice"){ print "Precision Machine Design" } else {print ""} \
# we skip the duplicate & empty columns for 37 & 38
if ($39 == "Intermediate" || $39 == "Advanced" || $39 == "Novice"){ print "Wind Turbine Design" } else {print ""} \
if ($40 == "Intermediate" || $40 == "Advanced" || $40 == "Novice"){ print "Mechanical Engineering" } else {print ""} \
# we skip 41, which is actually just the users gmail address
if ($42 == "Intermediate" || $42 == "Advanced" || $42 == "Novice"){ print "Agricultural Engineering" } else {print ""} \
if ($43 == "Intermediate" || $43 == "Advanced" || $43 == "Novice"){ print "Automotive Engineering" } else {print ""} \
if ($44 == "Intermediate" || $44 == "Advanced" || $44 == "Novice"){ print "Reliability Engineering" } else {print ""} \
if ($45 == "Intermediate" || $45 == "Advanced" || $45 == "Novice"){ print "Industrial Engineering" } else {print ""} \
if ($46 == "Intermediate" || $46 == "Advanced" || $46 == "Novice"){ print "Life Cycle Logistics" } else {print ""} \
if ($47 == "Intermediate" || $47 == "Advanced" || $47 == "Novice"){ print "Computer Programming" } else {print ""} \
if ($48 == "Intermediate" || $48 == "Advanced" || $48 == "Novice"){ print "Engineering" } else {print ""} \
if ($49 == "Intermediate" || $49 == "Advanced" || $49 == "Novice"){ print "Hydraulics / Pneumatics" } else {print ""} \
# this one is not like the others
if ($50 == "Yes"){ print "FreeCAD" } else {print ""} \
print "\t" $41 "\t" $51 "\t" "\n" \
}' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/\t,/\t/g' | \
sed 's/,\t/\t/g' | \
sed 's/^,//g'
exit 0
user@ose:~/tmp/phplist/listImport/designSprints$
- I added a new attribute to phplist, #18 = "How much time can you contribute to OSE per week?"
- there's a column in this spreadsheet for storing the user's non-primary email for google hangouts = "How much time can you contribute to OSE per week?"
- these days we use the open source software jitsi meet provided by Atlassian at https://meet.jit.si instead of Google Hangouts
- therefore, I think we should deprecate this column. Regarding data loss, we'd it looks like 88 out of 358 rows have a distinct email address for gmail than their primary email
user@ose:~/tmp/phplist/listImport/designSprints$ cat designSprints.1.csv | awk -F "\t" '{ if( $7!="" && $41!="") { if( $7 != $41){ print $7 " " $41}} }' | sort | uniq | wc -l
89
user@ose:~/tmp/phplist/listImport/designSprints$
- most of these aren't even email address; they just re-typed their First & Last name (for Google+ accounts I guess). Google+ is dead too; I'm killing this column.
- I confirmed that the spreadsheet on google docs hasn't been updated since the copy I downloaded last week-ish (so no data loss)
- I spent some time copying & pasting all of the options already imported into phplist into the script so that I was 100% sure there were no mismatches leading to data loss
user@ose:~/tmp/phplist/listImport/designSprints$ cat csvToPhplist.sh
#!/bin/bash
cat designSprints.2.csv | awk -F "\t" 'BEGIN {ORS=","} { \
# skip $1 = "timestamp", which we don:t need
# skip $4 = "action", which is "yes" or "subscribe" in every case
# skip $8 = "Google+ Name" which is an empty column
print $2 "\t" $3 "\t" $5 "\t" $6 "\t" $7 "\t" $9 "\t"
if ($10 == "Intermediate" || $10 == "Advanced" || $10 == "Novice"){ print "Project Management" } else {print ""} \
if ($11 == "Intermediate" || $11 == "Advanced" || $11 == "Novice"){ print "Community Management" } else {print ""} \
if ($12 == "Intermediate" || $12 == "Advanced" || $12 == "Novice"){ print "Electrical Engineering" } else {print ""} \
if ($13 == "Intermediate" || $13 == "Advanced" || $13 == "Novice"){ print "Computer Aided Design (CAD, CAM, CAE)" } else {print ""} \
if ($14 == "Intermediate" || $14 == "Advanced" || $14 == "Novice"){ print "Computer Animation / Modeling" } else {print ""} \
if ($15 == "Intermediate" || $15 == "Advanced" || $15 == "Novice"){ print "Video Production / Video Editing / Script Writing / Explainer Videos" } else {print ""} \
if ($16 == "Intermediate" || $16 == "Advanced" || $16 == "Novice"){ print "Technical Writing / Documentation" } else {print ""} \
if ($17 == "Intermediate" || $17 == "Advanced" || $17 == "Novice"){ print "Graphics / Design / Infographics / Computer Graphics" } else {print ""} \
if ($18 == "Intermediate" || $18 == "Advanced" || $18 == "Novice"){ print "Fabrication / Digital Fabrication" } else {print ""} \
# we skip a few empty columns
if ($23 == "Intermediate" || $23 == "Advanced" || $23 == "Novice"){ print "Machine Design" } else {print ""} \
if ($24 == "Intermediate" || $24 == "Advanced" || $24 == "Novice"){ print "Mechatronics" } else {print ""} \
if ($25 == "Intermediate" || $25 == "Advanced" || $25 == "Novice"){ print "Power Electronics" } else {print ""} \
if ($26 == "Intermediate" || $26 == "Advanced" || $26 == "Novice"){ print "Electrical Motor / Generator Design" } else {print ""} \
if ($27 == "Intermediate" || $27 == "Advanced" || $27 == "Novice"){ print "Electronics" } else {print ""} \
if ($28 == "Intermediate" || $28 == "Advanced" || $28 == "Novice"){ print "Hydraulic Motor Design" } else {print ""} \
# we skip the duplicate & empty column for 29
if ($30 == "Intermediate" || $30 == "Advanced" || $30 == "Novice"){ print "Industrial Laser Design" } else {print ""} \
if ($31 == "Intermediate" || $31 == "Advanced" || $31 == "Novice"){ print "Industrial Robotics Design" } else {print ""} \
if ($32 == "Intermediate" || $32 == "Advanced" || $32 == "Novice"){ print "Solar Engineering" } else {print ""} \
if ($33 == "Intermediate" || $33 == "Advanced" || $33 == "Novice"){ print "Metallurgy" } else {print ""} \
if ($34 == "Intermediate" || $34 == "Advanced" || $34 == "Novice"){ print "Hot Metal Processing" } else {print ""} \
if ($35 == "Intermediate" || $35 == "Advanced" || $35 == "Novice"){ print "Tool & Die" } else {print ""} \
if ($36 == "Intermediate" || $36 == "Advanced" || $36 == "Novice"){ print "Precision Machine Design" } else {print ""} \
# we skip the duplicate & empty columns for 37 & 38
if ($39 == "Intermediate" || $39 == "Advanced" || $39 == "Novice"){ print "Wind Turbine Design" } else {print ""} \
if ($40 == "Intermediate" || $40 == "Advanced" || $40 == "Novice"){ print "Mechnical Engineering" } else {print ""} \
# we skip 41, which is actually just the users gmail address
if ($42 == "Intermediate" || $42 == "Advanced" || $42 == "Novice"){ print "Agricultural Engineering" } else {print ""} \
if ($43 == "Intermediate" || $43 == "Advanced" || $43 == "Novice"){ print "Automotive Engineering" } else {print ""} \
if ($44 == "Intermediate" || $44 == "Advanced" || $44 == "Novice"){ print "Reliabilty Engineering" } else {print ""} \
if ($45 == "Intermediate" || $45 == "Advanced" || $45 == "Novice"){ print "Industrial Engineering" } else {print ""} \
if ($46 == "Intermediate" || $46 == "Advanced" || $46 == "Novice"){ print "Life Cycle Logistics" } else {print ""} \
if ($47 == "Intermediate" || $47 == "Advanced" || $47 == "Novice"){ print "Computer Programming / Database / CMS / Wiki" } else {print ""} \
if ($48 == "Intermediate" || $48 == "Advanced" || $48 == "Novice"){ print "Engineering" } else {print ""} \
if ($49 == "Intermediate" || $49 == "Advanced" || $49 == "Novice"){ print "Hydraulics / Pneumatics" } else {print ""} \
# this one is not like the others
if ($50 == "Yes"){ print "FreeCAD" } else {print ""} \
print "\t" $41 "\t" $51 "\n" \
}' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/\t,/\t/g' | \
sed 's/,\t/\t/g' | \
sed 's/^,//g'
exit 0
user@ose:~/tmp/phplist/listImport/designSprints$
- I created a new attribute in phplist, #20 = "Please link to your CV or LinkedIn
- I created a new list #5 in phplist = "Design Sprints"
- I imported the new awk'd csv into phplist. it added 294 new emails & subscribed 342 to the Design Sprints list
294 emails succesfully imported to the database and added to 1 lists. 351 emails subscribed to the lists 63 emails already existed in the database 6 Invalid Emails found. These records were added, but the email has been made up fromInvalid email [number] Subscriber data was updated for 357 subscribers 0 subscribers were matched by foreign key, 357 by emai
- Unfortunately, when I went to look at the user's data, I realized that one user, for example, had the following skills:
- CAE)
- CAM
- Computer Aided Design (CAD
- so it looks like my commas of the skill within a CSV broke it up into multiple fields. Worse yet, it created new options for all users.
- when I went back to edit the values for the skills attribute, it's clear which ones were added by the import, as they all have order = 0
- CAE)
- CAM
- Computer Aided Design (CAD
- Computer Programming / Database / CMS / Wiki
- the last one is really weird. I dug into the source code on the phplist wui just to see the differences.
<input type="text" name="listorder[44]" value="0" size="5" class="listorder" /> Computer Programming / Database / CMS / Wiki </div> ... <input type="text" name="listorder[37]" value="37" size="5" class="listorder" /> Computer Programming / Database / CMS / Wiki </div>
- so apparently one has two spaces after CMS and one has only one space
- I confirmed that the script only has one space, so it must be the original that accidentally had two spaces. I deleted the original (with the order = 37) and changed the new one (with order = 0) to now have order = 37.
- I went to delete the 3x duplicate CAD entries, but it complained that I couldn't because 18 subscribers had selected that value. It gave me the option to replace the value with another, so that solves the issue. I selected the real CAD option and deleted the "Computer Aided Design (CAD" one
- ugh, that didn't e an appear to work, though. Some users who had a subset of the CAD options didn't have the real CAD checkbox checked. I think I'll have to import again. I'll try with the comma escaped with a backslash..
- I did the import again
0 emails succesfully imported to the database and added to 1 lists. 3 emails subscribed to the lists 357 emails already existed in the database 6 Invalid Emails found. These records were added, but the email has been made up fromInvalid email [number] Subscriber data was updated for 357 subscribers 0 subscribers were matched by foreign key, 357 by email
- unfortunately, that created 3x new fields again; it didn't give a shit about the backslash
- CAE)
- CAM\
- Computer Aided Design (CAD\
- I think the easist solution is to make it a semicolon. we can probably migrate to a different one with a comma in the future, if needed
- I did this and imported again
0 emails succesfully imported to the database and added to 1 lists. 3 emails subscribed to the lists 357 emails already existed in the database 6 Invalid Emails found. These records were added, but the email has been made up fromInvalid email [number] Subscriber data was updated for 357 subscribers 0 subscribers were matched by foreign key, 357 by email
- that worked. I just deleted the original one with the comma (order = 4) and changed the order of the new one with semicolons from 0 to 4.
- I went to create a new subscribe page. I did one that allows users to subscribe to both the OSEmail newsletter *and* the Design Sprints newsletter at once. Unfortunately, the list description that I typed is public (I thought it was internal)
- I sent an email to Marcin linking to the new subscribe page, and I asked him to type up a public description for each list and send it to me.
- I created a copy of the Technical Team Culturing Survey before I clobber it https://docs.google.com/a/opensourceecology.org/forms/d/1ZTFyak2mtisyETp2LJogDgrj9XIOyacD2-sbsuzEHpI/edit
- I clobbered the existing form https://docs.google.com/forms/d/e/1FAIpQLSfefQ7vmZXhWZgMYz6pJB_GPxrqs2vurqBeQfdM_CJFJ1xUug/viewform
- fwiw, this was the final bash/awk script
user@ose:~/tmp/phplist/listImport/designSprints$ cat csvToPhplist.sh
#!/bin/bash
cat designSprints.2.csv | awk -F "\t" 'BEGIN {ORS=","} { \
# skip $1 = "timestamp", which we don:t need
# skip $4 = "action", which is "yes" or "subscribe" in every case
# skip $8 = "Google+ Name" which is an empty column
print $2 "\t" $3 "\t" $5 "\t" $6 "\t" $7 "\t" $9 "\t"
if ($10 == "Intermediate" || $10 == "Advanced" || $10 == "Novice"){ print "Project Management" } else {print ""} \
if ($11 == "Intermediate" || $11 == "Advanced" || $11 == "Novice"){ print "Community Management" } else {print ""} \
if ($12 == "Intermediate" || $12 == "Advanced" || $12 == "Novice"){ print "Electrical Engineering" } else {print ""} \
if ($13 == "Intermediate" || $13 == "Advanced" || $13 == "Novice"){ print "Computer Aided Design (CAD; CAM; CAE)" } else {print ""} \
if ($14 == "Intermediate" || $14 == "Advanced" || $14 == "Novice"){ print "Computer Animation / Modeling" } else {print ""} \
if ($15 == "Intermediate" || $15 == "Advanced" || $15 == "Novice"){ print "Video Production / Video Editing / Script Writing / Explainer Videos" } else {print ""} \
if ($16 == "Intermediate" || $16 == "Advanced" || $16 == "Novice"){ print "Technical Writing / Documentation" } else {print ""} \
if ($17 == "Intermediate" || $17 == "Advanced" || $17 == "Novice"){ print "Graphics / Design / Infographics / Computer Graphics" } else {print ""} \
if ($18 == "Intermediate" || $18 == "Advanced" || $18 == "Novice"){ print "Fabrication / Digital Fabrication" } else {print ""} \
# we skip a few empty columns
if ($23 == "Intermediate" || $23 == "Advanced" || $23 == "Novice"){ print "Machine Design" } else {print ""} \
if ($24 == "Intermediate" || $24 == "Advanced" || $24 == "Novice"){ print "Mechatronics" } else {print ""} \
if ($25 == "Intermediate" || $25 == "Advanced" || $25 == "Novice"){ print "Power Electronics" } else {print ""} \
if ($26 == "Intermediate" || $26 == "Advanced" || $26 == "Novice"){ print "Electrical Motor / Generator Design" } else {print ""} \
if ($27 == "Intermediate" || $27 == "Advanced" || $27 == "Novice"){ print "Electronics" } else {print ""} \
if ($28 == "Intermediate" || $28 == "Advanced" || $28 == "Novice"){ print "Hydraulic Motor Design" } else {print ""} \
# we skip the duplicate & empty column for 29
if ($30 == "Intermediate" || $30 == "Advanced" || $30 == "Novice"){ print "Industrial Laser Design" } else {print ""} \
if ($31 == "Intermediate" || $31 == "Advanced" || $31 == "Novice"){ print "Industrial Robotics Design" } else {print ""} \
if ($32 == "Intermediate" || $32 == "Advanced" || $32 == "Novice"){ print "Solar Engineering" } else {print ""} \
if ($33 == "Intermediate" || $33 == "Advanced" || $33 == "Novice"){ print "Metallurgy" } else {print ""} \
if ($34 == "Intermediate" || $34 == "Advanced" || $34 == "Novice"){ print "Hot Metal Processing" } else {print ""} \
if ($35 == "Intermediate" || $35 == "Advanced" || $35 == "Novice"){ print "Tool & Die" } else {print ""} \
if ($36 == "Intermediate" || $36 == "Advanced" || $36 == "Novice"){ print "Precision Machine Design" } else {print ""} \
# we skip the duplicate & empty columns for 37 & 38
if ($39 == "Intermediate" || $39 == "Advanced" || $39 == "Novice"){ print "Wind Turbine Design" } else {print ""} \
if ($40 == "Intermediate" || $40 == "Advanced" || $40 == "Novice"){ print "Mechnical Engineering" } else {print ""} \
# we skip 41, which is actually just the users gmail address
if ($42 == "Intermediate" || $42 == "Advanced" || $42 == "Novice"){ print "Agricultural Engineering" } else {print ""} \
if ($43 == "Intermediate" || $43 == "Advanced" || $43 == "Novice"){ print "Automotive Engineering" } else {print ""} \
if ($44 == "Intermediate" || $44 == "Advanced" || $44 == "Novice"){ print "Reliabilty Engineering" } else {print ""} \
if ($45 == "Intermediate" || $45 == "Advanced" || $45 == "Novice"){ print "Industrial Engineering" } else {print ""} \
if ($46 == "Intermediate" || $46 == "Advanced" || $46 == "Novice"){ print "Life Cycle Logistics" } else {print ""} \
if ($47 == "Intermediate" || $47 == "Advanced" || $47 == "Novice"){ print "Computer Programming / Database / CMS / Wiki" } else {print ""} \
if ($48 == "Intermediate" || $48 == "Advanced" || $48 == "Novice"){ print "Engineering" } else {print ""} \
if ($49 == "Intermediate" || $49 == "Advanced" || $49 == "Novice"){ print "Hydraulics / Pneumatics" } else {print ""} \
# this one is not like the others
if ($50 == "Yes"){ print "FreeCAD" } else {print ""} \
print "\t" $51 "\n" \
}' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/g' | \
sed 's/,,/,/tboylasarg' | \
sed 's/,,/,/g' | \
sed 's/\t,/\t/g' | \
sed 's/,\t/\t/g' | \
sed 's/^,//g'
exit 0
user@ose:~/tmp/phplist/listImport/designSprints$
Mon Dec 17, 2018
- I checked the files in the ose-server-backups bucket on backblaze b2. The 34G file is hidden, staged for deletion. The past 2 days' daily backups are 17.5G. So I think I resolved the last issue of backups including other backups from '/home/b2user/sync*/
- The current bucket size is 86G with 12x files and growing
- The billing section shows only a $0.04 charge for "B2 - Estimated Upcoming Charges" Storage fes between "11/16 - 12/16" (so far)
- I guess we'll be on-track to finish this migration to b2 on 2019-01-01, a good date to document the change from dreamhost to backblaze b2.
- ...
- Marcin said the base64-encoded email test of the repermission campaign from phplist _still_ had image display issues (note that the old URL I recorded on my wiki log is now a 200, not a 404 anymore). Fuck it, I don't think we can keep playing these games with google. I think the only solution is to actually attach the images to the email. Hopefully that works OK
- the relevant documentation was basically useless https://www.phplist.org/manual/ch022_creating-a-template.xhtm[[1]]
- I came across this old phplist form post talking about the option EMBEDUPLOADIMAGES https://forums.phplist.com/viewtopic.php?t=39994
- And I found the relevant documentation page about this config option, which appears to do what we want. Note that it should make the images in the *template* attached, but not necessarily the images in the campaign's CONTENT. In any case, our repermission campaign has no images in its campaign content, so this will at least let us get the initial repermission campaign out
- I added the following lines to our phplist config at /var/www/html/phplist.opensourceecology.org/config.php
// attach images because otherwise gmail MITMs our links and causes 404s
define('EMBEDUPLOADIMAGES',1);
- I sent a test campaign and confirmed that the source now included base64 sections for all 5x images (fb like, tweet, fwd, ose logo, and phplist logo) as well as the main body of the message in base64. This seems like the most robust solution
- note that this change to attached images made the email grow from 65K to 119K. Not too bad. Honestly, as long as it's under 500K, I'm pretty happy.
- ...
- Marcin said that it's not really necessary to collect percise information about _how_ skilled someone is on the Tech Culturing Survey data. In many cases, they're not an accurage gauge of this anyway. So, instead, we can greatly simplify the attributes in phplist by just having them check-off skills/expertise that they have. Simple: check or no check.
- This is actually _extremely_ easy with phplist's checkboxgroup attribute type. We can actually have a single attribute and add sills to it over time. The example used in this documentation is "which foods do you like?" perfect! https://resources.phplist.com/documentation/configuration
- I converted the spreadsheet to a csv and used some bash magic to extract the skills we want
user@ose:~/tmp/phplist/listImport/designSprints$ head -n1 designSprints.1.csv | tr "\t" "\n" | cut -d[ -f2 | sed 's/]//g' Timestamp Name First Name (Given Name) Are you subscribing or unsubscribing to the Design Sprints list? City, State, Country Formatting (no abbreviations please) - ex (strict): Maysville, Missouri, United States / Barcelona, Spain By filling out this form, you agree to have your name and skill set published openly on our wiki so others can collaborate with you. How much time can you commit per week? Preferred Email address Google+ name Phone number (with country code): Project Management Community Management Electrical Engineering Computer Aided Design (CAD, CAM, CAE) Computer Animation / Modeling Video Production / Video Editing / Script Writing / Explainer Videos Technical Writing / Documentation Graphics / Design / Infographics / Computer Graphics Fabrication / Digital Fabrication Computer Animation / Modeling Videography / Video Editing / Script Writing Computer Programming Community Management Machine Design Mechatronics Power Electronics Electrical Motor / Generator Design Electronics Hydraulic Motor Design Hydraulic Motor Design Industrial Laser Design Industrial Robotics Design Solar Engineering Metallurgy Hot Metal Processing Tool & Die Precision Machine Design Precision Machine Design Solar Engineering Wind Turbine Design Mechnical Engineering If Gmail is not your preferred email, please enter you Gmail address so we can connect via Google Hangout.(If applicable) Agricultural Engineering Automotive Engineering Reliabilty Engineering Industrial Engineering Life Cycle Logistics Computer Programming / Database / CMS / Wiki Engineering Hydraulics / Pneumatics Are you proficient with FreeCAD? Please paste a link to your resume or email it to info@opensourceecology.org user@ose:~/tmp/phplist/listImport/designSprints$
- I created a new phplist attribute #16 = "Please identify which skills you possess" with the following checkboxes
Project Management Community Management Electrical Engineering Computer Aided Design (CAD, CAM, CAE) Computer Animation / Modeling Video Production / Video Editing / Script Writing / Explainer Videos Technical Writing / Documentation Graphics / Design / Infographics / Computer Graphics Fabrication / Digital Fabrication Computer Animation / Modeling Videography / Video Editing / Script Writing Computer Programming Community Management Machine Design Mechatronics Power Electronics Electrical Motor / Generator Design Electronics Hydraulic Motor Design Hydraulic Motor Design Industrial Laser Design Industrial Robotics Design Solar Engineering Metallurgy Hot Metal Processing Tool & Die Precision Machine Design Precision Machine Design Solar Engineering Wind Turbine Design Mechnical Engineering Agricultural Engineering Automotive Engineering Reliabilty Engineering Industrial Engineering Life Cycle Logistics Computer Programming / Database / CMS / Wiki Engineering Hydraulics / Pneumatics FreeCAD
- Attempting to add these resulted in an error on the phplist wui
Database error 1062 while doing query Duplicate entry 'Computer Animation / Modeling' for key 'name' Database error 1062 while doing query Duplicate entry 'Community Management' for key 'name' Database error 1062 while doing query Duplicate entry 'Hydraulic Motor Design' for key 'name' Database error 1062 while doing query Duplicate entry 'Precision Machine Design' for key 'name' Database error 1062 while doing query Duplicate entry 'Solar Engineering' for key 'name' Add new Please identify which skills you possess, one per line
- I guess those were just superfluious duplicates *shrug*
- It looks like now I can import thise data into phplist by using {TAB} as the main delimiter and an actual comma as the delimiter for the new checkbox group. This will require some logic to achieve https://discuss.phplist.org/t/importing-an-user-with-multiple-values-in-the-same-checkboxgroup-attribute/1568/5
- I did a quick POC for how we can do this in awk. It will be painful, but it should work
user@ose:~/tmp/phplist/listImport/designSprints$ cat designSprints.1.csv | awk -F "\t" '{print $10}' | head
Please identify your skill set and rate your level of expertise, or mark "not applicable." [Project Management]
Intermediate
Intermediate
Intermediate
Intermediate
dae
Advanced
Novice
Intermediate
Not Applicable
user@ose:~/tmp/phplist/listImport/designSprints$ cat designSprints.1.csv | awk -F "\t" '{if ($10 == "Intermediate" || $10 == "Advanced" || $10 == "Novice"){ print "Project Management," } else {print ""} }' | head
Project Management,
Project Management,
Project Management,
Project Management,
Project Management,
Project Management,
Project Management,
user@ose:~/tmp/phplist/listImport/designSprints$
- I sent an email to Marcin asking what should be checked and unchecked. For example, if a user marked their skill as "Novice," should this result in a check next to this skill for them or not?
- I also created a new attribute #17 = "Which workshops have ou attended?". I also made this a checkboxgroup. We didn't already have this attribute in our old spreadsheets, but the other tabs in the Design Sprints spreadsheets have this data implicitly (there's a tab, for example, called
Sun Dec 16, 2018
- I could find no way to limit the textline attribute to be integer-only, so I posted to the phplist discourse forums about the best way to migrate our 32x skills questionaire into phplist here https://discuss.phplist.org/t/possible-to-limit-data-type-in-textline-attributes/4776
- Spent some time reading through the phplist documentation on list segmentation as we may want to use this feature to send specific campaigns based on user's skills https://www.phplist.org/manual/ch018_targeting-your-campaigns.xhtml
- Sent Marcin an email about this dillema, and was clear that I want to be careful about how we design these attributes in phplist to prevent pain in the long-term. I asked him if he's confident that the existing 4x options for the attributes ("Not Applicable", "Novice", "Intermediate", and "Advanced") would be OK for the next few years or decades, and I sent him info on List Segmentation.
Sat Dec 15, 2018
- Marcin sent me an email that the latest repermssion campaign test had broken images as well.
- I checked my email in chromium via gmail.com, and I confirmed the images were broken
- The image href location was this google proxy shit that mitm'd our email's contents. Indeed, hitting this URL results in a 404
- Dissecting the URL above, we can see the original image URL =
- hitting this URL loads the image successfully. Therefore, the issue is definitely on Google's end. Da fuk, google? Why you break our image content with your shitty proxy?
- The image href location was this google proxy shit that mitm'd our email's contents. Indeed, hitting this URL results in a 404
- I sent myself a new campaign from the same draft repermission campaign, and I confirmed that the image is viewable in thunderbird but broken in gmail (in chromium)
- I checked the "view source" in gmail, and I saw this for the image
<td align=3D"center" bgcolor=3D"#6a9afc" class=3D"w640" i= d=3D"header" style=3D"font-family:'Helvetica Neue', Arial, Helvetica, Genev= a, sans-serif;border-collapse:collapse;" width=3D"640"> =09=09=09=09=09=09<div align=3D"right" style=3D"text-align:right;"><img ali= gn=3D"top" border=3D"0" class=3D"w640" id=3D"customHeaderImage" label=3D"He= ader Image" src=3D"https://phplist.opensourceecology.org/uploadimages/osema= il_header_logo_2012.png" style=3D"display:inline;height:auto;line-height:10= 0%;outline-style:none;text-decoration:none;" width=3D"639" height=3D"198" a= lt=3D"OSE Header Logo" title=3D"OSE Header Logo"/></div> =09=09=09=09=09=09</td>
- I checked the same block in thunderbird, and got this
<td align=3D"center" bgcolor=3D"#6a9afc" class=3D"w640" i= d=3D"header" style=3D"font-family:'Helvetica Neue', Arial, Helvetica, Genev= a, sans-serif;border-collapse:collapse;" width=3D"640"> =09=09=09=09=09=09<div align=3D"right" style=3D"text-align:right;"><img ali= gn=3D"top" border=3D"0" class=3D"w640" id=3D"customHeaderImage" label=3D"He= ader Image" src=3D"https://phplist.opensourceecology.org/uploadimages/osema= il_header_logo_2012.png" style=3D"display:inline;height:auto;line-height:10= 0%;outline-style:none;text-decoration:none;" width=3D"639" height=3D"198" a= lt=3D"OSE Header Logo" title=3D"OSE Header Logo"/></div> =09=09=09=09=09=09</td>
- So the contents is exactly the same. Note that the double-quotes are a bit weird: 3D"
- I checked the contents within the campaign template within phplist; it's a bit different
<td align="center" bgcolor="#6a9afc" class="w640" id="header" style="font-family:'Helvetica Neue', Arial, Helvetica, Geneva, sans-serif;border-collapse:collapse;" width="640"> <div align="right" style="text-align:right;"><img align="top" alt="OSE Header Logo" border="0" class="w640" height="198" id="customHeaderImage" label="Header Image" src="https://phplist.opensourceecology.org/uploadimages/osemail_header_logo_2012.png" style="display:inline;height:auto;line-height:100%;outline-style:none;text-decoration:none;" title="OSE Header Logo" width="639" /></div> </td>
- I'm thinking that this issue could be caused by a very long line being broken up, and that break is breaking the gmail image proxy. The solution may be to `fold` the html so that it doesn't
- Another solution is to change the "Content-Transfer-Encoding" from "quoted-printable" to "base64". Personally, I fucking love base64 encoding to avoid these sorts of issues, but I've read that it can trigger spam flags, so I'll look at that as a last resort.
- The URL alone is too fucking long. We want it to be 75 characters long, but our url alone is 79 characters :\
https://phplist.opensourceecology.org/uploadimages/osemail_header_logo_2012.png
- Ok, so I tested base64 with the HTMLEMAIL_ENCODING option https://resources.phplist.com/system/config/htmlemail_encodingf
- I added the following lines to the end of the config.php file
// send base64 to prevent the contents from being mangled
define("HTMLEMAIL_ENCODING", "base64");
- After changing the encoding to base64, I sent another test campaign. This time the images displayed properly. Unfortunately, at the same time, the previously broken campaign in my inbox was fixed! So I'm not really sure if the issue is fixed or not. I did, howoever, confirm that the source of the email shows only base64 encoding.
- I sent another campaign to Marcin and asked him to validate if the images are now showing
- when I pass the entire base64-encoded email to `base64 --decode` and extract the same header image block as above, I get the original as desired; no shitty double-quotes replaced or awkward newlines
<td align="center" bgcolor="#6a9 afc" class="w640" id="header" style="font-family:'Helvetica Neue', Arial, Helvet ica, Geneva, sans-serif;border-collapse:collapse;" width="640"> <div align="right" style="text-align:right;"><img align="top" border="0" class="w640" id="customHeaderImage" label="Header Image" src="https://phplist.opensourceecology.org/uploadimages/osemail_header_logo_2012.png" style="display:inline;height:auto;line-height:100%;outline-style:none;text-decoration:none;" width="639" height="198" alt="OSE Header Logo" title="OSE Header Logo"/></div> </td>
- If this fails too, we may want to consider attaching the images directly to the email
- ...
- I added new attributes for the Design Sprints list
- City, attribute #11
- State/Provence, attribute #12
- Country, attribute #13
- Phone Number, attribute #14
Fri Dec 14, 2018
- I logged into our backblaze account, and now I see that the latest daily backup is 34G. That's much bigger than I expected!
- I double-checked the other file, and found that it's 17G & from today (2018-12-14)
[root@hetzner2 backups]# date Fri Dec 14 14:57:19 UTC 2018 [root@hetzner2 backups]# du -sh /root/backups/sync.old/* 17G /root/backups/sync.old/hetzner2_20181214_072001.tar.gpg [root@hetzner2 backups]#
- I checked the backup2.sh script, and I found an issue. I'm actually _missing_ the mysql dump! I went ahead and uncommented that line too, so the next backup will be actually bigger..
- I checked the file on the backblaze b2 wui; it's distinct than the above file. The above 17G file is 'hetzner2_20181214_072001.tar.gpg' but the b2 file is 'daily_hetzner2_20181214_092018.tar.gpg'
- so the 17G file is probably the backup.sh script's file sent to dreamhost. The 34G file in b2 was actually deleted
- ah! I forgot that the b2user executes the b2 binary because we don't trust it. Therefore, the new sync directory is in /home/b2user/sync
[root@hetzner2 backups]# du -sh /home/b2user/sync/* 32G /home/b2user/sync/daily_hetzner2_20181214_092018.tar.gpg [root@hetzner2 backups]# du -sh /home/b2user/sync.old/* 16G /home/b2user/sync.old/daily_hetzner2_20181213_091827.tar.gpg [root@hetzner2 backups]#
- so it went from 16G to 32G. Duh, it's backing up the old backup. Let's add it to the exclude
- original line in backup2.sh for backing up /home/
# /home/
$MKDIR "${backupDirPath}/${archiveDirName}/home"
time $NICE $TAR -czf ${backupDirPath}/${archiveDirName}/home/home.${stamp}.tar.gz /home/*
- new line to exclude the /home/user/b2user/sync* dirs
# /home/
$MKDIR "${backupDirPath}/${archiveDirName}/home"
time $NICE $TAR --exclude "${b2StagingDir}*" -czf ${backupDirPath}/${archiveDirName}/home/home.${stamp}.tar.gz /home/*
- to be safe, I went ahead and truncated the existing backup file in /home/b2user/sync/ so that the next couple day's backups won't explode
[root@hetzner2 sync]# du -sh daily_hetzner2_20181214_092018.tar.gpg 32G daily_hetzner2_20181214_092018.tar.gpg [root@hetzner2 sync]# truncate -s0 daily_hetzner2_20181214_092018.tar.gpg [root@hetzner2 sync]# du -sh daily_hetzner2_20181214_092018.tar.gpg 0 daily_hetzner2_20181214_092018.tar.gpg [root@hetzner2 sync]#
- I'm still waiting for Marcin's response about the Technical Team Survey import into phplist, so I began looking at better let's encrypt renewal process
- previously, there was an issue with a subset of domains failing causing the entire process to fail. I found an option '--allow-subset-names', which changes this default behaviour
--allow-subset-of-names When performing domain validation, do not consider it a failure if authorizations can not be obtained for a strict subset of the requested domains. This may be useful for allowing renewals for multiple domains to succeed even if some domains no longer point at this system. This option cannot be used with --csr. (default: False)
- the existing update script is pretty bare
[root@hetzner2 ~]# cat /etc/cron.d/letsencrypt # once a month, update our letsencrypt cert 20 4 13 * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log [root@hetzner2 ~]# cat /root/bin/letsencrypt/renew.sh #!/bin/bash /bin/certbot renew /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* /sbin/service nginx reload # exit cleanly exit 0 [root@hetzner2 ~]#
- I updated the script to to include the new option listed above and to send an email if the `certbot` command returns a non-zero execution status
[root@hetzner2 ~]# cat /root/bin/letsencrypt/renew.sh #!/bin/bash /bin/certbot renew --allow-subset-of-names # alert when there's an error from the renewal attempt if $0 ; then # the `certbot` execution returned a non-zero exit status echo "The attempt to renew your letsencrypt certificates with certbot failed" | mail -s 'certbot fail' letsencrypt@opensourcecology.org fi /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* /sbin/service nginx reload # exit cleanly exit 0 [root@hetzner2 ~]#
- Unfortunately when I tested the `mail` command, I didn't get anything forwarded to my email address. So I probably need to setup email forwarding from the letsencrypt@opensourceecology.org account
- I spent like 15 minutes failing to login to this account. I have the fucking password, but Google says I need to validate with a phone number. I don't have a phone number god dammit! In another tab, I'm fucking logged in as the admin. I clicked around on the user's profile, but I don't see any locks in-place. Nothing for override. https://admin.google.com/ac/users/26in1rg0ml5as9
- There's no useful info here https://support.google.com/accounts/answer/7162782
- I did more digging into why the ossec emails are emtpy; I confirmed that it's not a gpg issue
[root@hetzner2 ossec]# pwd
/var/ossec
[root@hetzner2 ossec]# cat sent_encrypted_alarm.sh
#!/bin/bash
# store the would-be plaintext email body
plaintext=`/usr/bin/formail -I ""`
# loop through all recipients & send them individually-encrypted mail
recipients="michael@opensourceecology.org marcin@opensourceecology.org"
for recipient in $(echo $recipients); do
# leave nothing unencrypted, including the subject!
echo "${plaintext}" | /usr/bin/gpg --homedir /var/ossec/.gnupg --trust-model always -ear "${recipient}" | /usr/bin/mail -r noreply@opensourceecology.org -s "" "${recipient}"
done
exit 0
[root@hetzner2 ossec]# echo "test" |/usr/bin/gpg --homedir /var/ossec/.gnupg --trust-model always -ear "michael@opensourceecology.org"
gpg: WARNING: unsafe ownership on homedir `/var/ossec/.gnupg'
-----BEGIN PGP MESSAGE-----
Version: GnuPG v2.0.22 (GNU/Linux)
hQIMA/VBynCfrWvvAQ//VauxWW82yd/8OVe8E224I8OXY6b+YUsp2/jDhixXIw3o
lR5sljY785yIo2/wg5yXRnDlBAD3i+G5nnQwK4NpLHq5mJkCHPp10/iaU793l+dw
fYQI0v4tDgFWD7c5/tSDIsPVgsT3tGWX2E/KU5qcfz/HTa8yRzHstdZBeSQvT2rR
SqJn7RoVigHMpDY7KBRbJZdz8Tu3fMZ3eCKXpuliNAZ7t0PyegwvLMjKj0yrj/vx
8mP1gKjlUF/dl7jXohwbDICZShbuAezEaWn5uUvf7bldCrWfoq46NkGV9J2I9Gqo
keRWzsvPhj8S/kLB/hZtoBADXQMZPRX+QaX7ZopcEZbkiNSwoNOPgltjVA+tCkT6
a5lbSq+tSDuTpCa6OhJUFjbhzQHDPr/g6YpwCAk/GI17R7SmKqn/y+QnOXaaOdBi
hHPLiMwWF4vNcZhxm69boLtzGHHcxjVACvOVvUX9yq4W+hi7lzBl+lvrh+/oaa8N
ukxMHjf+gmNYBfI6NR7WFKsV8GRQtt+pnQ6DZj2lbquxuKHAkxRuuf5vMTXSgj+3
I2b3BXuvuHHxMigIq7ioR3zbZeOGXyWH81fkp1LT4BIUeKVbRTnUFrS7NdOGqhOA
ex38NDAUyRXVmoOLTyIzX5syy5XnL5i1sXIDGh4B9G4wpU6dzZk+4/F7hJASKjHS
QAGFWVGFeNA33WCB8aPiopwUBkHIOL5yPinSVDzO08u4EshjYpvd2rgGvVv3omOn
sxxu5JpjLGx1x2fnxPJ5MKQ=
=CehV
-----END PGP MESSAGE-----
[root@hetzner2 ossec]#
- I checked the ossec logs and found something interesting
2018/04/03 22:00:49 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:03:04 rootcheck: INFO: Ending rootcheck scan. 2018/04/03 22:06:52 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:12:52 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:18:54 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:24:57 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:30:59 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:36:28 INFO: Connected to 127.0.0.1 at address 127.0.0.1, port 25 2018/04/03 22:37:00 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:43:02 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:49:04 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 22:55:07 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:01:09 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:07:09 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:13:12 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:19:14 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:25:16 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:31:17 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:37:19 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:43:22 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:49:24 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/03 23:55:12 ossec-syscheckd: INFO: Starting syscheck scan. 2018/04/03 23:55:26 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/04 00:01:27 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/04 00:07:29 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/04/04 00:08:29 ossec-syscheckd: INFO: Ending syscheck scan. ... 2018/12/14 14:46:24 INFO: Connected to 127.0.0.1 at address 127.0.0.1, port 25 2018/12/14 14:47:41 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 14:53:44 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 14:59:46 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:05:46 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:11:49 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:17:51 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:23:53 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:29:54 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:35:56 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:41:58 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:48:01 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 15:52:12 INFO: Connected to 127.0.0.1 at address 127.0.0.1, port 25 2018/12/14 15:54:03 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 16:00:04 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 16:06:06 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 16:10:32 INFO: Connected to 127.0.0.1 at address 127.0.0.1, port 25 2018/12/14 16:12:08 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 16:18:11 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry 2018/12/14 16:24:11 ossec-analysisd: ERROR: read error on /queue/diff/hetzner2/535/last-entry
- These read errors on the last-entry file have been occuring since 2018-04-04
- Found an article about this suggesting that the solution is to stop ossec, delete the problematic last-entry file, and start ossec https://support.google.com/accounts/answer/7162782
- the restart was successful, and the errors didn't pop-up on restart. I got an email, but it was still empty :(
- I noticed that the encryption had a WARNING that the homedir's permissions were wrong. I tried to reset them
[root@hetzner2 ossec]# [root@hetzner2 ossec]# ls -lah /var/ossec/.gnupg total 104K drwx------ 2 ossec root 4.0K Dec 14 16:24 . dr-xr-x--- 15 root ossec 4.0K Dec 14 16:40 .. -rwx------ 1 ossec root 7.5K Jul 10 2017 gpg.conf -rwx------ 1 root root 40K May 30 2018 pubring.gpg -rwx------ 1 root root 40K May 30 2018 pubring.gpg~ -rw------- 1 ossec ossec 600 Dec 14 16:24 random_seed -rwx------ 1 ossec root 0 Jul 10 2017 secring.gpg -rwx------ 1 ossec root 1.2K May 30 2018 trustdb.gpg [root@hetzner2 ossec]# chmod -R 0700 /var/ossec/.gnupg [root@hetzner2 ossec]# chown -R ossec:root /var/ossec/.gnupg [root@hetzner2 ossec]# ls -lah /var/ossec/.gnupg total 104K drwx------ 2 ossec root 4.0K Dec 14 16:24 . dr-xr-x--- 15 root ossec 4.0K Dec 14 16:40 .. -rwx------ 1 ossec root 7.5K Jul 10 2017 gpg.conf -rwx------ 1 ossec root 40K May 30 2018 pubring.gpg -rwx------ 1 ossec root 40K May 30 2018 pubring.gpg~ -rwx------ 1 ossec root 600 Dec 14 16:24 random_seed -rwx------ 1 ossec root 0 Jul 10 2017 secring.gpg -rwx------ 1 ossec root 1.2K May 30 2018 trustdb.gpg [root@hetzner2 ossec]#
- I couldn't get the error to go away, but I think it's an issue with root/ossec. In any case, it's a warning not an error. I am beginning to think this is actually an issue with `formail`, not gpg
Wed Dec 12, 2018
- Marcin responded stating that the images weren't loading on the phplist repermission campaign within gmail. I confirmed that the images aren't loading in gmail (they are loading in thunderbird, though). What the fuck is gmail doing?
- I came across an article suggesting that the issue was missing alt/title/width/height attributes https://stackoverflow.com/questions/21437916/html-image-not-showing-in-gmail
- I tried editing the mail template to add the height/alt/title attributes to the main header image. I resent a test campagn, and I found that *all* the images were fixed. I don't consider that a sufficient test & fix, though. Indeed, I didn't make any chanes to the social media images (fb/twitter/fwd), but they were fixed as well! Maybe it's never broken, except for forwards? I can't do much to debug google's issues. I just sent another test campaign to marcin directly & asked if he sees the image this time..
- ...
- I began looking at migrating the Deisgn Sprints = Techincal Team Culturing spreadsheet into phplist
- This document has several tabs. The first is just survey results from this google form https://docs.google.com/forms/d/e/1FAIpQLSfefQ7vmZXhWZgMYz6pJB_GPxrqs2vurqBeQfdM_CJFJ1xUug/viewform
- There's a tab called "Team Backhoe" with just 4 subscribers with first name, last name, email, and phone number fields
- There's a tab called "Mailchimp import" that has 200 rows, many of which are corrupt. The fields are email address, first name, and last name, but there's actually no data--just references to the first tab in the spreadsheet = Survey Results
- There's a tab called "Sheet2" with 124 rows of subscribers' first names, last names, email addresses, and phone numbers.
- There's a tab called "Team Ironworker" this tab has no entries at all. Not even columns. It's entirely blank.
- So the Survey Results tab in the spreadsheet is by far the most complicated. In the form it's very simple, but to get the data for a user's experience with 31x different skillsets between None/Novice/Intermediate/Advanced will be non-trivial to setup in phplist
- I did a test to create a new attribute with type = "select" (a drop-down menu)
- I set the attribute name to "How would you identify your level of expertise with Hydraulics / Pneumatics?", the type to "select", and added the following fields to the attribute's drop-down selection options: "None\nNovice\nIntermediate\nAdvanced".
- I updated the poc subscribe page to include this new test attribute, and it worked as desired. So this _does_ work, but it'll be a pain to add each of the attributes and each of the items to each of the attributes. And if we want to change the possible items, it's worse. Perhaps we should just ask people to rate their experience between 0-10?
- ...
- I logged back into our backblaze account.
- I saw 2 current daily backups plus 2 more that are hiden
- We can't make hidden files actually delete, it seems, so this is as good as we can get it.
- Today is the 12th. At this time, I see backups from today (the 12th), and yesterday (the 11th). I also have the ability to delete the 2 hidden (staged for deletion) files via the wui from 3 and 4 days ago (the 9th & 8th) note that, oddly, there's no file from the 10th. hmm.
- Anyway, that's sufficent for the "oh fuck, I deleted a super-important file; let me grab it from the most recent backups asap". It's decent for "oh crap, the website is down due to some corruption and I didn't notice it for a day." It begins to be an issue if the issue wasn't noticed for a few days...
- I see 3x monthlys from 2018-10-01, 2018-11-01, and 2018-12-01
- That's perfect. So, no matter what, it looks like we'll have at most 31 days of data loss. It's less than ideal, but it works for our current non-existant budget
- I also see 3x weekly uploads from 2018-11-26, 2018-12-03, and 2018-12-10 (all on Mondays).
- I didn't initially intend to have these, but it may not be a bad idea.
- I saw 2 current daily backups plus 2 more that are hiden
- so, all in all, we have 10x files. Currently they're just 11.3M each, but soon they'll be 17G each. That'll be 170G. But as the monthlys grow, that'll probably be 13 monthlys + 4 dailys + 5 weeklys (at most?). And let's assume 20G dailys = 22 * 20G = 440G
- I re-ran the backbaze b2 calculator with
- initial upload 440G
- upload 20G * 31 days = 620G per month
- monthly delete of 180G
- monthly download of 10G
- it came out to $199.20
- removing the weeklys, that's maybe 13+4 = 17 * 20G = 340G. All-in-all, recalulations dropped it to $193.20. I think I'll leave-in the weeklys; we don't save much by not doing so. Most of the costs here come from the uploaded bytes ($3.10). The storage costs are actually fairly cheap ($1.70).
- I actually think this number is high and there's insufficient fiels to describe our process. We'll just have to see how the first month goes.
- I uncommented the lines in the /root/backups/backup2.sh script to include the following directires going forward (in addition to just /etc/ before)
- /home/
- /var/log/
- /root/
- /var/www/
Mon Dec 10, 2018
- I began parsing the data in the Workshop Interests Mapping spreadsheet genearted from the survey https://docs.google.com/forms/d/1tI6Opcu2bZKoRFlPoLHyI7lWHTEBcSlhn9QkQCAGzAA/viewform
- I requrested edit access to the above form
- I began doing some crazy bash shit to isolate the possible entries for each field, but then I discovered the form; better to copy from the original form than to try to extract a subset of options
user@ose:~/tmp/phplist/listImport/workshopInterest$ cut -f3 workshopInterest.2.csv > workshops.txt user@ose:~/tmp/phplist/listImport/workshopInterest$ cat workshops.txt | tr "," "\n" | sed -e 's/^:space:*//' | sort | uniq 3D Printer Aquaponic Greenhouse CEB Press CNC Circuit Mill CNC Torch Table FreeCAD and How to Design Things Ironworker Machine Laser Cutter MicroCar Open Source Microtractor Other Plastic Recycling to 3D Printing Filament Seed Eco-Home What workshop are you interested in? user@ose:~/tmp/phplist/listImport/workshopInterest$
- from the form, we'd need a couple new attributes with checkboxes
- workshop interests with 13x choices
3D Printer CNC Circuit Mill Laser Cutter Plastic Recycling to 3D Printing Filament CEB Press Open Source Microtractor Seed Eco-Home Aquaponic Greenhouse CNC Torch Table FreeCAD and How to Design Things Ironworker Machine MicroCar Other
- Outcome Interests
Extreme Manufacturing - experience one day swarm builds with a team of people as a social production process where you build your practical skills Products -- taking home machines that you can use, modify, and truly own Continuing Education or Professional Development - for librarians, teachers, or anyone else who would like to broaden their horizons Curriculum - for teaching classes, home schooling, and other training materials Enterprise Traininig - starting a small business producing parts, household goods, houses, 3D printers, tractors, and more Team Building - developing collaborative skills, improving teamwork, improving morale of your team, corporate team building retreats, Fun - a social event intended for relaxation or just doing something different with a good group of people Changing the World - meeting people who have a civic vision of collaboration for the common good to bring about the open source economy OSE Clubs - Start an OSE Club at your school and get involved in world-changing parallel design Other
- Note here that the "Other" checkbox actually has a input box next to it. I don't think phplist would let us do that, so we'd probably need to have an additional field called "Is there anything else you want to get out of your workshop experience?"
- here the sed came in handy so I could see the manually entrys in the "other" field
user@ose:~/tmp/phplist/listImport/workshopInterest$ cat outcomes.txt | tr "," "\n" | sed -e 's/^:space:*//' | sort | uniq 3D printers A full blown non-profit OSE hackerspace for the city as well. and more and other training materials and truly own Changing the World - meeting people who have a civic vision of collaboration for the common good to bring about the open source economy college Continuing Education or Professional Development - for librarians corporate team building retreats Curriculum - for teaching classes Enterprise Traininig - starting a small business producing parts Extreme Manufacturing - experience one day swarm builds with a team of people as a social production process where you build your practical skills Fun - a social event intended for relaxation or just doing something different with a good group of people hands-on training experiences? home schooling household goods houses improving morale of your team improving teamwork I would like to create a club (MeetUp) that is not affiliated with any school modify or anyone else who would like to broaden their horizons or university. Want to avoid accreditations and bureacracy as much as possible. OSE Clubs - Start an OSE Club at your school and get involved in world-changing parallel design Products -- taking home machines that you can use teachers Team Building - developing collaborative skills tractors What would you like to get out of our immersive user@ose:~/tmp/phplist/listImport/workshopInterest$
- there's a text-input comment field on the field as well. Probably we should name this attribute "Any other comments or suggestions?" in phplist to match the form. Here's what we got back so far in the spreadsheet:
user@ose:~/tmp/phplist/listImport/workshopInterest$ cut -f7 workshopInterest.2.csv | sort | uniq A 3D printer workshop in Prince George might be the most practical way to begin! :) Any other comments or suggestions? Been admiring your work for years, I want to share with the back to the land people here in Nova Scotia. can i be an ambassador? I would like to sponsor a workshop in my area (pay for and keep the machine have people learn aws its built) C.I.F.P Pico Frentes CNC mill would be cool. Double post but with coordinates to avoid confusion Everythink you do is great ! Keep going. Glad to host a Workshop at our School Have a STEM group locally; with dedicated workspace. I also believe we need a social network & market alternative ( making one ) to connect all local networks into a global realtime super network, similar to what the global market spans currently. Also, love you guys!! ♥ I have a sustainable urban location where you could have the workshop. https://www.joinerylbc.org and https://www.restorationcollectivetulsa.org I love the work you guys and gals are doing. Keep it up! :) I'm not an educator in a strict sense. I'm a member of a team (of architects, designers an a programmer) that is setting up a hub for creative industries in Timisoara, Romania. This hub will host a makespace among many other things. We are very interested in the implementation of open source "dynamics". Education will be one of our pillars. Another way of designing, fabricating and consuming much more meaningfull and less harmfull is possible. I represent a group in Copenhagen called Flydende By. We're a sort of radical warehouse of people that build stuff and do community activities. We have about 3500 likes on Facebook and a decent network in the Danish capital. We could potentially host and feed teachers as well as the workshops themselves. With enough prior warning we could even gather a decent amount of materials for re-use :) I speak french, english and spanish. I can move about in europe to get to a workshop I've been making some Arduino and RPi projects. Next up, FreeCAD I would like to also say I can travel up to the bay area. Please keep me posted as I will go to the one in the bay area if it is on the right days. I yeach at the Advance Manufacturing Center and would love to have one of your workshops at our center. Let's see how this survey works. Looking forward to it.... Love you guys! Keep it up! May not be able to lead a club for a year or two but very interested in starting one. Would like to build the large tractor and nickel iron battery as well as other machines. None at this time. Please come to Europe! It would be super dope! :D Synchronizing Open Source Organizations Thanks for your work Thank you for your efforts We can host workshop events and have some expirience with that. Our open workshop is well equiped - all kinds of welding, metalwork, electronics etc. Its also possible to accomodate people at the same place. user@ose:~/tmp/phplist/listImport/workshopInterest$
- other attributes to add are:
- city
- state/provence
- country
- I noticed that our microfactory website actually aggrigates the coordinates/country inputs of this form in the spreadsheet and plots it on a google map here https://microfactory.opensourceecology.org/request-a-workshop/
- Without non-trivial development, I wouldn't expect this to be possible after we deprecate the spreadsheet and migrate to phplist. I sent Marcin an email about what he expected to happen to the map after we moved to phplist with this survey
Sun Dec 09, 2018
- Marcin asked me if we could change the text in the phplist subscribe page
- "I prefer to receive emails in Text format" to "I prefer to receive emails in Text-only format (no graphics)", but I saw no way to change this in the wui (either general settings or the specific "Subscribe Page" settings)
- "Subscribe to Selected Newsletters" should just be "Subscribe"
- I should look into this more. It may be better if we have just 1 real subscribe page--so long as it doesn't break the ajax form. Anyway, I have to finish creating the other lists and their cooresponding subscribe pages first before I can test this.
- Marcin pointed out that the subscribe page states that required forms are written in red, but none are written in red! This is probably a consequence of me destroying/overriding the default CSS colors https://phplist.opensourceecology.org/lists/?p=subscribe&id=3
- note that this unbranded subscribe page does show text in red https://phplist.opensourceecology.org/lists/?p=subscribe&id=3
- I fixed this by adding a ".required{ color: #990000; }" block to the css styles tag of the header. The new header for the subscribe page is
<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />
<style>
html, body {
width: 100%;
margin: 0 auto;
}
.phplist_subscribe_header {
background-color: #E1E1E1;
width: 100%;
height:121px;
background-image: url('/uploadimages/800px-OSE_logo_2014-grey-1.png');
background-repeat: no-repeat;
}
.phplist_subscribe_footer {
clear: both;
width: 100%;
padding: 20px;
margin: 0 auto;
color: #bbb;
border-top: none;
margin: 0 auto;
overflow: hidden;
background-color: #3D484A;
}
.phplist_subscribe_footer a {
color: #9c9e9e;
}
.required {
color: #990000;
}
</style>
</head>
<body>
<div class="phplist_subscribe_header">
<span> </span>
</div>
<div id="wrapper" class="container">
<div id="mainContent">
<div class="panel">
<br />
<div class="content well">
Thr Dec 06, 2018
- Marcin granted me access to the OSEmail and Design Sprints google docs forms that populate their cooresponding subscriber google docs spreadsheets
- I created a backup of the OSEmail form here https://docs.google.com/forms/d/1GC3R0DfdK7UObXMZjD5XN5mVtHhggLqyFGuUFZjtx08/edit
- I deleted all the inputs on the original OSEmail form, and replaced the top message with a deprecation notice, redirecting the user back to our wiki--which is the source-of-truth with a link for how to subscribe (it now links to the phplist subscribe page) https://docs.google.com/forms/d/1XXJjAa6be9uiuviMo7kqdwZNGTNtU8DjeVvve-aMieY/viewform
- the new form still allows submissions, but it ends up just dropping a timestamp row and all the other fields are empty. should be no issue here.
- the one user that "subscribed" to the (now deprecated) OSEmail spreadsheet actually typed an invalid email address; I'll ignore them.
- ...
- I spent some time trying to make the phplist site make more legitimate with ose branding
- I fabricated a header & footer by taking the look & feel from the microfactory & osemain websites
<style>
html, body {
width: 100%;
margin: 0 auto;
}
.phplist_subscribe_header {
background-color: #E1E1E1;
width: 100%;
height:121px;
#background-image: url('./800px-OSE_logo_2014-grey-1.png');
background-image: url('https://microfactory.opensourceecology.org/wp-content/uploads/2018/09/800px-OSE_logo_2014-grey-1.png');
background-repeat: no-repeat;
}
.phplist_subscribe_footer {
clear: both;
width: 100%;
padding: 20px;
margin: 0 auto;
color: #bbb;
border-top: none;
margin: 0 auto;
overflow: hidden;
background-color: #3D484A;
}
.phplist_subscribe_footer a {
color: #9c9e9e;
}
</style>
<div class="phplist_subscribe_header">
<span> </span>
</div>
<p>content</p>
<div class="phplist_subscribe_footer">
<div id="footer-wrap">
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/deed.en_US"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-sa/4.0/88x31.png"></a>
<br/><br/>
<span>All Open Source Ecology content on this website</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/deed.en_US">Creative Commons Attribution-ShareAlike 4.0 International License</a>.</span>
</div>
- this replaces the existing header ("Config" -> "Settings" -> "subscription-ui settings" -> "Header of public pages")
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" /> <meta name="theme-color" content="#2C2C2C"/> <link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" /> <link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" /> <link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" /> </head> <body class="fixed invisible"> <div id="container"> <div id="header" class="navbar navbar-inverse"> <div id="rack-functions" class="container"> <h1 id="logo"><a href="http://[WEBSITE]" title="Visit our website">[ORGANISATION_NAME]</a></h1> </div> </div> <div id="wrapper" class="container"> <div id="mainContent"> <div class="panel"> <br /> <div class="content well">
- and the existing footer ("Config" -> "Settings" -> "subscription-ui settings" -> "Footer of public pages")
</div>
</div>
</div><!-- ENDOF #mainContent-->
</div><!-- ENDOF .wrapper -->
</div><!-- ENDOF #container -->
<div id="footer">
<div id="footerframe">
<ul class="list-unstyled">
<li> </li>
</ul>
</div>
</div>
<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="admin/js/phplistapp.js"></script>
<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<script>
/* do not remove this from here */
$(document).ready(function(){
if ( $('body').hasClass('invisible') ){
myfunction();
}
$('#edit_list_categories input.form-control').tagsinput('refresh');
});
</script>
</body>
</html>
- I encountered a modsecurity false-positive when attempting to update the footer
- whitelisted 958030, xss
- I encountered a modsecurity false-positive when attempting to update the footer
- I also uploaded our ose logo from the microfactory site and uploaded it to phplist. we should now be able to use "[LOGO]" in our templates to reference it
- unfortuatnely, nothing is changing after I changed the header & footer contents!
- it looks like the actual code that's being displayed is coming from the public_html/lists/admin/ui/dressprow/ directory https://github.com/phpList/phplist-ui-dressprow
- I found a relevant discussion on this here https://forums.phplist.com/viewtopic.php?f=36&t=39453
- It was suggested that I have to change the settings for the subscribe-page-specific header ("Config" -> "Subscribe pages" -> osemail -> edit)
- here's the relevant Header before my changes
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" /> <meta name="theme-color" content="#2C2C2C"/> <link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" /> <link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" /> <link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" /> </head> <body class="fixed invisible"> <div id="container"> <div id="header" class="navbar navbar-inverse"> <div id="rack-functions" class="container"> <h1 id="logo"><a href="http://phplist.opensourceecology.org" title="Visit our website">[ORGANISATION_NAME]</a></h1> </div> </div> <div id="wrapper" class="container"> <div id="mainContent"> <div class="panel"> <br /> <div class="content well">
- and the footer before my changes
</div>
</div>
</div><!-- ENDOF #mainContent-->
</div><!-- ENDOF .wrapper -->
</div><!-- ENDOF #container -->
<div id="footer">
<div id="footerframe">
<ul class="list-unstyled">
<li> </li>
</ul>
</div>
</div>
<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="admin/js/phplistapp.js"></script>
<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<script>
/* do not remove this from here */
$(document).ready(function(){
if ( $('body').hasClass('invisible') ){
myfunction();
}
$('#edit_list_categories input.form-control').tagsinput('refresh');
});
</script>
</body>
</html>
- that appeared to have worked, but it would be nice if it was more universal, not requiring setup for each list
- I modified the header & footer a bit so that the default styles for the content were still present (with nice padding, boxing, background color, spacing, etc for the form inputs)
- new header
<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />
<style>
html, body {/uploadimages/800px-OSE_logo_2014-grey-1.png
width: 100%;
margin: 0 auto;
}
.phplist_subscribe_header {
background-color: #E1E1E1;
width: 100%;
height:121px;
background-image: url('/uploadimages/800px-OSE_logo_2014-grey-1.png');
background-repeat: no-repeat;
}
.phplist_subscribe_footer {
clear: both;
width: 100%;
padding: 20px;
margin: 0 auto;
color: #bbb;
border-top: none;
margin: 0 auto;
overflow: hidden;
background-color: #3D484A;
}
.phplist_subscribe_footer a {
color: #9c9e9e;
}
</style>
</head>
<body>
<div class="phplist_subscribe_header">
<span> </span>
</div>
<div id="wrapper" class="container">
<div id="mainContent">
<div class="panel">
<br />
<div class="content well">
- new footer
</div>
</div>
</div>
</div>
<div class="phplist_subscribe_footer">
<div id="footer-wrap">
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/deed.en_US"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-sa/4.0/88x31.png"></a>
<br/><br/>
<span>All Open Source Ecology content on this website</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/deed.en_US">Creative Commons Attribution-ShareAlike 4.0 International License</a>.</span>
</div>
</body>
</html>
- I began the work to import the True Fans list into phplist
- I realized that this data should really be in a CRM. I sent Marcin an email asking for his general thoughts about CRM at OSE.
- I created a new list called True Fans did *not* make it public. It's id = 4
- I created two new attributes
- 7 = donor level
- 8 = First Name (I'll just put the full names here, and we can have new users divide this up properly going forward)
- 9 = Last Name
- After a *lot* of manual & lossy cleanup, I successfully imported 983 out of 1,120 subscribscribers into the new phplist True Fans list
983 emails succesfully imported to the database and added to 1 lists. 1025 emails subscribed to the lists 137 emails already existed in the database 60 Invalid Emails found. These records were added, but the email has been made up fromInvalid email [number] Subscriber data was updated for 1120 subscribers 0 subscribers were matched by foreign key, 1120 by email
Wed Dec 05, 2018
- I checked the google doc & found that we had one more OSEmail signup yesterday (2018-12-04), even though I tried deprecating it on the wiki article https://wiki.opensourceecology.org/wiki/OSEmail
- I tried searching for the form url on search engines, but couldn't dig up another page linking to the old subscription form. I requested permission to edit the form via google docs from the onwer = marcin https://docs.google.com/forms/d/1XXJjAa6be9uiuviMo7kqdwZNGTNtU8DjeVvve-aMieY
- I logged into phplist and confirmed that nobody had signed up since I migrated to it. Hmm, no idea how the user found the deprecated google form. Once Marcin grants me permission, I'll just change the form to a dumb message that says it's deprecated and provide a link back to the wiki article at https://wiki.opensourceecology.org/wiki/OSEmail#Subscribe
- ...
- I began to import the True Fans spreadsheet into phplist, but I quickly found that the data is very corrupt
- first, there's no column titles
- there's email addresses in several different columns
- there's an occassional field with value = 'quit'. Is this supposed to be the "Unsubscribe" equivalent?
- there's occassional fields with dates, but--without any context--we may just want to ignore these fields
- there's some fields with a dolar value; probably we should ignore these as well
- there's some fields with gold, angel, platnum, quit, Gold Extra, etc. This is fairly consistant, but it jumps around columns
- of the consistant data, I think we can clean up the encoding and import the following fields: name, email address, donor status (gold, platnum, angel, etc),
- Unfortunately, because there's no dates, I have no idea when the last entry was added. But I did download a snapshot of the data from today (2018-12-05), so I can do a csv diff of a future download to see if the data is actively being modified or not
- I sent an email to marcin to clairify this data before I attempt to clean it up (a lossy process) and import it into phplist
- ...
- I began looking at the spreadsheet for the "Design Sprints" = "OSE Technical Team Culturing Survey" https://docs.google.com/spreadsheets/d/176mESULRogtjzNi-C0yn3Yue3f9trIW_O9dNum89pfE
- I changed the above document to be private = only viewable to specific users it's shared with (previously it was public!)
- I also changed the sharing of the OSEmail spreadsheet to be private (previously it was viewable to anyone at the OSE org with a link)
- I also changed the sharing of the True Fans spreadsheet to be private (previously it was viewable to anyone at the OSE org with a link)
- I confirmed that the Workshop Interest spreadsheet is already private
- I scanned the wiki article titled "OSE Design Sprint" https://wiki.opensourceecology.org/wiki/OSE_Design_Sprint
- and found a link to the suvey that populates this spreadsheet https://wiki.opensourceecology.org/wiki/Tech_Team_Culturing_Survey
- the actual survey iframed is here https://docs.google.com/forms/d/e/1FAIpQLSfefQ7vmZXhWZgMYz6pJB_GPxrqs2vurqBeQfdM_CJFJ1xUug/viewform?embedded=true&formkey=dGNONk5LOUVLNXZQUE1COGp4Z1hUV2c6MQ
- I couldn't find a button to request access to this form
- oh, I guess I don't see a button to request edit access because I already have it! I found the form via "form" -> "Edit form" in the google docs menu. I confirmed that cannot edit the OSEmail form (I get a "Request edit access" link in the top-right of the page instead)
- I confirmed that there does not appear to be a form tied to the "True Fans" spreadsheet. Instead of a "form" menu (which is absent), I do see a "Tools" -> "Create a form" menu option
- I confirmed that there *is* a form tied to the "Workshop Interest Mapping" spreadsheet, but I don't have edit permission
- I sent an email to Marcin explicitly asking permission for me to edit the google docs forms for the "OSEmail" and "Workshop Interest Mapping" spreadsheets
Mon Dec 03, 2018
- my query about requiring the user to explicitly accept the ToS during a repermission campaign recieved no replys https://discuss.phplist.org/t/re-permission-campaign-requiring-explicity-tos-action/4710
- I think we'll just have to proceed by incluing a link to the Privacy Policy in the body of the repermission campaign, and make the big green button say "I agree to the Privacy Policy" on the first line.
- ...
- Marcin sent me a link to his suggested body of the repermission campaign https://wiki.opensourceecology.org/wiki/OSEmail_Repermissioning_Campaign#Repermissioning_Email
- I made many updates, including html-ifying it, basic grammer changes, and removing some techincal inaccuracies
- I began to import the OSEmail spreadsheet users into phplist
- Note that as soon as I download the spreadsheet's contents, it's essentially deprecated. I did this today (2018-12-03), and I see that the last subscriber was yestereday (2018-12-02) and there were two more subscribers 2 days ago (2018-12-03). So after I get this thing imported into phplist, I should deprecate the google form and replace it with the ajax one
- note that the google spreadsheet is pretty dumb; it just has a ton of rows with an action = subscribe or unsubscribe. So, I have to manually find all the entries with 'unsubscribe' in the row, then remove the cooresponding email address line. Fortunately, there was only
- I hit some issues with the commas as the delimter, since the "how did you hear about us?" field has many commas in it. libreoffice csv export will wrap a field in quotes to prevent this, but I'm not sure that phplist supports this encoding. phplist does default to tab-delimited-values for "csv" though — so I just did that instead of commas
- In the end, I imported 1,156 subscribers into the "OSEmail" group. I'm not sure the output makes total sense--I guess there were duplicates in the rows
1053 emails succesfully imported to the database and added to 1 lists. 1098 emails subscribed to the lists 103 emails already existed in the database 1 Invalid Emails found. These records were added, but the email has been made up fromInvalid email [number] Subscriber data was updated for 1156 subscribers 0 subscribers were matched by foreign key, 1156 by email
- I attempted to add the ajax form to the wiki, but it didn't work — probably due to javascript complications in the wiki
- I sent an email to marcin asking which websites & pages we should put the ajax form
Tue Nov 29, 2018
- Marcin asked me to remove the address of OSE from our OSEmail template. No problem, but I reminded him that it's listed in many places all over the web https://wiki.opensourceecology.org/widki/OSEmail#2018-09
- Had a call with Marcin, talked about discourse, phplist, and backblaze
- Marcin recovered his passprhase for his ssh private key
- Marcin was successfully able to mount the ose shared keepass file that lives on the hetzner2 ose server onto his local machine via sshfs
- Marcin was able to decrypt the ose shared keepass file and gain access to our ose backblaze b2 credentials that I stored in the shared ose keepass file
- Marcin was able to add the 2FA secret key for our backblaze b2 account into his phone's Google Authenticator app
- Marcin was able to login to our backblaze b2 account using the user/pass in keepass + his google auth 2fa
- Marcin was able to view the contens of our ose-server-backups bucket in the backblaze b2 wui
- Marcin was able to add our billing information into our backblaze b2 account
- Success!! Now I finally can migrate our backups from dreamhost onto backblaze b2
Wed Nov 28, 2018
- sent an email to Marcin asking him to draft a new OSEmail for our upcoming repermission campaign
- I read through this guide showing examples of repermission campaigns. my take aways: https://econsultancy.com/gdpr-examples-repermissioning-emails-campaigns/
- We should make it as obvious as possible that the user must opt-in or (if they do nothing) they will be permanently blacklisted & receive no future newsletters
- The subject should very clearly indicate that their action is required. I recommend: "Opt-In Action Required for OSEmail" or something similar
- Rather than put the opt-in confirmation link as a hyperlink in the body of the text, we should actually make a huge green button titled "Yes, keep sending me the OSEmail newsletter" or something similar
- The top of the email should be about the opt-in. The normal OSEmail update should go below the big green button
- I spent some time crafting the above big green button html + fallback plaintext. Here's what I have so far
Hello friends, <br/><br/> Update from OSE: We have just setup phplist for our OSEmail newsletter. If you would like to keep recieving emails, you must <a href="http://phplist.opensourceecology.org/lists/?p=confirm&uid=43d035a41d2d893d21255158de6f9831">click this link</a>. <a href="[CONFIRMATIONURL]" style="display:block; text-decoration:none; color: #FFFFFF;"> <div style="text-align:center; margin: 50px"> <span style="padding:15px; width: 100%; background-color: #03C03C; border-radius:15px; font-size: large; font-weight: bold;"> Yes, keep sending me the OSEmail newsletter! </span> </div> </a> <br/><br/> If you cannot see the link above, you can copy and paste the below link into your browser to keep recieving the OSEmail newsletter: <br/><br/> [CONFIRMATIONURL] </div> blah blah blah. some shit we did this past month. blah blah blah <br/><br/> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent nec mauris purus. Curabitur id lectus ornare, feugiat metus ut, porttitor nunc. Maecenas pharetra aliquet dictum. Vivamus vel vulputate ligula, non aliquam dolor. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Vestibulum finibus libero sed augue ornare, vel venenatis ligula porttitor. Integer sodales sed nisl eu suscipit. Nunc gravida vehicula sem, a congue massa venenatis eu. Morbi convallis urna a felis tincidunt, rutrum elementum sem varius. Vestibulum congue lectus nec ex venenatis sagittis. Quisque sed orci vitae mauris ultricies dignissim vel sit amet tortor. Aliquam imperdiet maximus enim, at posuere metus eleifend aliquet. Duis convallis ligula urna, eget hendrerit quam blandit at. <br/><br/> In eleifend porttitor quam eget sollicitudin. Vivamus egestas pellentesque nisi, sit amet porta nunc lobortis a. Proin interdum mi eu mauris aliquam ultricies. Sed lobortis sapien leo, porttitor vehicula purus molestie id. Morbi quis nunc id enim egestas efficitur pharetra sagittis tellus. Praesent imperdiet dolor urna, et porttitor eros dictum sit amet. Sed at fringilla turpis. Vivamus rhoncus porttitor massa, vitae venenatis lorem sagittis ac. Quisque quis erat sed magna elementum aliquet id non justo. Mauris cursus quam sed elit sodales, a pellentesque lorem euismod. Donec sodales sem non enim mattis volutpat. In auctor tincidunt posuere. Praesent posuere lectus quis elit accumsan, id congue ipsum euismod. Fusce molestie luctus metus, eu malesuada urna. Vivamus gravida dignissim porttitor. Nam sed diam efficitur, feugiat nulla id, tincidunt augue. <br/><br/> Nulla facilisi. Nunc vitae lobortis diam. Cras diam enim, finibus quis consequat finibus, rutrum et nisl. Morbi eleifend blandit dui, et eleifend mi viverra eget. Mauris ut congue massa. Aliquam erat volutpat. Suspendisse potenti. Praesent finibus ullamcorper ante, posuere consectetur neque bibendum eu. Sed ac sapien vitae augue blandit mattis. Pellentesque tincidunt sagittis mi at blandit. Sed tincidunt iaculis enim, volutpat accumsan felis imperdiet luctus. Cras metus leo, auctor ac purus sed, ornare mollis dui. Duis lobortis, libero id facilisis ullamcorper, tortor ipsum posuere purus, vel pellentesque ipsum urna eu nisl. <br/><br/> Quisque in ultricies mauris. Phasellus vel ante purus. Cras vel vestibulum ligula, vitae suscipit purus. Praesent at vehicula erat, varius blandit tellus. Vestibulum a mi vel lacus porttitor vestibulum in eu dolor. Duis eleifend purus iaculis massa tristique fermentum. Etiam vel velit nec nisi bibendum varius at viverra sapien. <br/><br/> In at tellus accumsan, egestas metus id, tempor nisi. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Vivamus eleifend elit dolor, vestibulum consectetur elit feugiat eget. Integer sed rutrum ante. Vivamus blandit neque sit amet mi pretium faucibus. Proin lectus ex, fringilla nec mi accumsan, maximus mattis turpis. In hac habitasse platea dictumst. Etiam sed tincidunt magna. Nullam purus turpis, tempor at elementum a, blandit nec massa.
- ...
- I wasn't quite sure if the reason why I wasn't required to explicitly check the box agreeing to the Privacy Policy was because the user already existed. So, I deleted (not just blacklisted) myself from the phplist subscribers db entirely, then imported -> repermission campaign -> click confirmation url.
- Still, I was unblacklisted without requiring to explicilty accepting the Privacy Policy. This is a huge issue!
- I created a topic on the phplist discourse forms asking if there's some way to require the user to explicitly accept the ToS after the ToS https://discuss.phplist.org/t/re-permission-campaign-requiring-explicity-tos-action/4710
Tue Nov 27, 2018
- I creado ated a new list in phplist called "OSEmail"
- I checked the box to make it a "Public list"
- this became list #3
- it looks like attributes are configured seperately, so after creating the new list, I went to "Config" -> "Configure attributes" -> "Add new Attribute" to add a new field called "How did you find out about us?"
- type = "textline"
- I unchecked "Is this attriubte required?"
- this became attribute #6
- ok, so now I guess it's the "subscribe pages" that ties it all together. I created a new Subscribe Page with title = "Subscribe to our OSEmail Newsletter"
- I left the default for "HTML Email choice" = "Offer checkbox for text"
- I left the default for "Display email address confirmation field" = "Display email address confirmation field"
- I left all the options in the "Transaction messages" tab at the defaults
- In the "Select the attributes to use" tab, I checked the "Check this box to use this attribute in the page" for these attributes:
- attribute 4 = I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.
- attribute 6 = How did you find out about us?
- In the "Select the lists to offer" tab, I checked only the box "OSEmail" list that I created above
- this created the following subscribe page https://phplist.opensourceecology.org/lists/?p=subscribe&id=3
- I validated that the OSEmail subscriber list is currently empty by the phplist wui @ "Subscribers" -> "Subscriber lists" -> "OSEmail" https://phplist.opensourceecology.org/lists/admin/?page=members&id=3
- I copied the phplistAjax.php file to see if I could successfully alter it to work for this new subscriber list (page)
- unfortunately, after changing the list's id from 2 to 3, it doesn't work...back to painful tracing
- ..
- There's this guide to using xdebug for tracing https://devzone.zend.com/1135/tracing-php-applications-with-xdebug/
- I tried & failed to do this back in August https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q3#Wed_Aug_22.2C_2018
- I tried enabling xdebug.autotrace & setting xdebug.trace_output_dir in the vhost's apache config file, but no trace logs appeared
- I tried adding the xdebug_start_trace() and xdebug_stop_trace() directly to the lists/index.php file, and still no trace logs were written
- fucking xdebug, I give up again.
- ...
- I changed the "Display email address confirmation field" radio button to "Don't display email address confirmation field", and the ajax form worked again.
- so here's the updated ajax form page
<html><body>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>
<script type="text/javascript">
function checkform() {
// first, clear the response div from the previous attempts results
jQuery("#result").empty();
for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
if (document.phplistSubscribeForm.elements[fieldstocheck[i]].checked) {
} else {
jQuery("#result").empty();
alert("The following field is required: "+fieldnames[i]);
eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
} else {
if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
}
}
for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
return false;
}
}
if (! checkEmail()) {
alert("Email address is not valid");
return false;
}
return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name) {
fieldstocheck[fieldstocheck.length] = value;
fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name) {
groupstocheck[groupstocheck.length] = value;
groupnames[groupnames.length] = name;
}
function compareEmail() {
return (document.phplistSubscribeForm.elements["email"].value == document.phplistSubscribeForm.elements["emailconfirm"].value);
}
function checkEmail() {
var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.phplistSubscribeForm.elements["email"].value);
}
function checkGroup(name,value) {
option = -1;
for (i=0;i<document.phplistSubscribeForm.elements[name].length;i++) {
if (document.phplistSubscribeForm.elements[name][i].checked) {
option = i;
}
}
if (option == -1) {
alert ("Please enter your "+value);
return false;
}
return true;
}
function submitForm() {
// first, clear the response div from the previous attempts results
jQuery("#result").empty();
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#phplistSubscribeForm').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
dataType: 'html',
<!-- set the id in the url below (&id=X) to the Subscribe List ID, per
the "Config" -> "Subscribe pages" page in the phplist wui
* https://phplist.opensourceecology.org/lists/admin/?page=spage
Note that the Subscribe Page should use
* "Don't display email address confirmation field" or it will fail
* "Select the lists to offer" tab should check exactly one list
For more information, see
* https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
-->
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=3',
// defines a function that's called when our ajax call suceeds (ie: gets a
// 200 response back)
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
// defines a function that's called when our ajax call fails (ie: gets a 500
// response back)
error: function (request, status, error) {
jQuery("#result").empty();
alert('Sorry, we were unable to process your subscription.');
}
});
}
</script>
<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">
<p>Signup for our newsletter!</p>
<!-- this simple form intentionally only requires the bare necessities:
[1] email address
[2] accept Privacy Policy
-->
<form method="post" action="" name="phplistSubscribeForm" id="phplistSubscribeForm">
<!-- [1] email address -->
<input type=text name=email required="required" placeholder="Email" size="30" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
<br/>
<!-- [2] accept Privacy Policy -->
<input type="checkbox" name="attribute4" value="on" class="attributeinput" id="attribute4" />
<label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a></label>
<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE Privacy Policy");</script>
<br/>
<!-- other strange hidden inputs per phplist's ajax example thread
* https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
-->
<input type="hidden" name="list[2]" value="signup" />
<input type="hidden" name="listname[2]" value="newsletter"/>
<input type="hidden" name="htmlemail" value="1" />
<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>
</form>
<!-- do some input sanity checking with js, then ajax() submit using jquery -->
<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>
</div>
<!-- this input will fill-in with results from the phplist server after the ajax
submission via jquery -->
<div id="result"></div>
<script type="text/javascript">
// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';
</script>
</body></html>
- ...
- I did a test import of users to the new list = OSEmail by the wui via "Subscribers" -> "Import subscribers" -> "Import by uploading a CSV file with email addresses and additional data.
- ironicly, this procedure to import a "CSV" has a default delimiter of a TAB. I had to change this to an actual comma (,)
- After clicking the "Import" button, I was prompted with a drop-down menu to match the columns to attributes. I used the column title "referrer" and selected the cooresponding attriubte name "How did you find out abou..."
- Here's the data I used:
user@personal:~/tmp/ose/phplist$ cat subscribeListTest.20181127.csv email,referrer michael@opensourceecology.org,my friend travis phplist@opensourceecology.org,marcin's ted talk from 2011 user@personal:~/tmp/ose/phplist$
- after I confirmed the import, I got the following email from phplist@opensourceecology.org
0 emails succesfully imported to the database and added to 1 lists. 2 emails subscribed to the lists 2 emails already existed in the database Subscriber data was updated for 2 subscribers 0 subscribers were matched by foreign key, 2 by email
- I noted that the imported subscribers did *not* recieve an email stating that they were added. This is what we want at this time, but not generally what we want when, for example, we manually import a list of subscribers from a sign-up sheet (data entry from a workshop or talk)
- ...
- I went ahead and clicked the "Enable" button for the plugin = "inviteplugin" via the phplist wui at "Config" -> "Mange plugins"
- I confirmed that both the subscribers imported above were not blacklisted and were subscribed to the OSEmail list
- I sent a new campaign, where I set the "Send as" = "Invite" (as opposed to "HTML" or "Text") on the Formait tab
- After I processed the repermission campagin, I got the email for both email accounts.
- I re-checked the subscribers, and I found that they were both blacklisted now (the OSEmail list was still listed as subscribed, but the blacklist overrides this)
- Without clicking the confirmation link in the repermission campaign's email, I tried to send another campaign to OSEmail. I confirmed that neither of the 2x subscribers (still blacklisted) recieved the follow-up campaign.
- I went ahead and clicked the confirmation link in one of the email providers http://phplist.opensourceecology.org/lists/?p=confirm&uid=43d035a41d2d893d21255158de6f9831
- When I loaded the above URL, I got a "Thank you for confirming your subscription to our newsletter..." message
- Also, after clicking the link, I got a "Welcome to our Newsletter" email
- I checked the subscriber again, and now the user wasn't listed as blacklisted
- digging deeper, I clicked the "History" button on the subscriber's profile, and the "Subscription" tab had the following entries
- 27 November 2018 20:33:53 Import by maltfield: http://phplist.opensourceecology.org/lists/admin/?page=user&id=2 No user details changed
- 27 November 2018 20:44:58 Added to blacklist: Added to blacklist for reason Blacklisted by the invitation plugin
- 27 November 2018 21:11:15 Removed from blacklist: Removed from blacklist
- 27 November 2018 21:11:15 Confirmation: Subscriber removed from do-not-send list for manual confirmation of subscription
- 27 November 2018 21:11:15 Confirmation: Lists: *newsletter *OSEmail
- therefore, we're making note (even for existing users) of when:
- we imported the subscriber into phplist (and which user did it)
- we blacklisted them via the invite plug as part of the repermission campaign to ensure gdpr compliance
- when the user clicked the link requesting to confirm their subscription. It also explicilty lists the URL of their confirmation link, ie: REQUEST_URI = /lists/?p=confirm&uid=43d035a41d2d893d21255158de6f9831
- note that I wasn't forced to check a box agreeing to the new Priacy Policy--which is the whole point of the re-permission campaign! This must be fixed!
Mon Nov 26, 2018
- discovered that our letsencrypt certifcate is expired on the prod site! It is now " November 26, 2018, 4:41" and it expired on " November 26, 2018, 4:41".
- we should probably have an automated alert go out when the cert is about to expire in less than 7 days.
- the log at /var/log/letsEncryptRenew.log shows errors from the phplist domain. This is probably a result of the port change from 4443 to 443.
- I updated the letsencrypt config file for the opensourceecology.org domain
[root@hetzner2 cron.d]# grep phplist /etc/letsencrypt/renewal/opensourceecology.org.conf phplist.opensourceecology.org = /var/www/html/certbot/htdocs [root@hetzner2 cron.d]# vim /etc/letsencrypt/renewal/opensourceecology.org.conf [root@hetzner2 cron.d]# grep phplist /etc/letsencrypt/renewal/opensourceecology.org.conf phplist.opensourceecology.org = /var/www/html/phplist.opensourceecology.org/public_html [root@hetzner2 cron.d]#
- I re-ran the cron job, and that fixed the websites
[root@hetzner2 cron.d]# cat /etc/cron.d/letsencrypt # once a month, update our letsencrypt cert 20 4 13 * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log [root@hetzner2 cron.d]# /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log [root@hetzner2 cron.d]#
Sat Nov 24, 2018
- I fixed the phplist email template overflow issue by appending 'width: 600px; overflow: hidden;" to the div in the td surrounding the [CONTENT] https://wiki.opensourceecology.org/wiki/File:OSEmail_phplist.tar.gz
- ...
- I sent Marcin an email about adding the billing info to our Backblaze B2 account on 10/20. It's been over a month, so I sent Marcin a follow-up again--asking if we should schedule a call so I can assist him in this task.
- I logged into our backblaze b2 account to check the status of the file uploads (currently just /etc/ as we're space limited with our free account)
- it looks like the data retention policy is only partially working properly: the monthly files are being kept as desired, but we currently have 8 daily files. This confirms my confusion from 4 months ago, when I first setup the lifecycle settings https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q3#Sat_Jul_28.2C_2018
I created the above lifecycle settings in the wui. It was easy, after logging in, just click on "buckets" on the left navigation panel, then click the "Lifecycle Settings" link on the corresponding bucket (ie: ose-server-backups). Then there's an easily dialog that opens. Click the "Use custom lifecycle rules" option, and "Add Lifecycle Rules", which will add another row to the list of rules. Each rule has 3 fields: "File Path (fileNamePrefix)", "Days Till Hide (daysFromUploadingToHiding)", and "Days Till Delete (daysFromHidingToDeleting)". I set both fields to the numbers as described above. I'm not sure if this means that daily files will be deleted in 3 or 6 days (the sum of the two?). I'll just check back later and see. Note that it wouldn't let me set either of these fields to "0".
- interstingly, 4/8 of the files that show in the wui don't show from the cli. also, there's no weekly files at all. hmm. here's a list of all the files currently on the server
[b2user@hetzner2 B2_Command_Line_Tool]$ ~/virtualenv/bin/b2 ls ose-server-backups daily_hetzner2_20181121_091749.tar.gpg daily_hetzner2_20181122_091803.tar.gpg daily_hetzner2_20181123_091819.tar.gpg daily_hetzner2_20181124_091828.tar.gpg monthly_hetzner2_20181001_091809.tar.gpg monthly_hetzner2_20181101_091810.tar.gpg [b2user@hetzner2 B2_Command_Line_Tool]$
- I added a stub Backblaze B2 section next to our glacier section in the OSE_Server wiki page https://wiki.opensourceecology.org/wiki/OSE_Server#Restore_from_Backblaze_B2
- I confirmed that I can download the "hidden" files from the wui, which were missing from the cli. I think this "Days Till Hide" = daysFromUploadingToHiding *should* be set to 0
- as mentioned before, this can't actually be set to 0. Also, it looks like these lifecycle settings cannot be updated. So I went ahead and deleted the existing "daily_" lifecycle rule with "daysFromUpladingToHiding=3 + daysFromHidingToDeleting=3" to "daysFromUpladingToHiding=2 + daysFromHidingToDeleting=1" via the wui. That should give us the data retention of 3 daily backups at a time, but I'll verify this in the future.
- I also went ahead and updated the lifecycle settings for "weekly_" and "monthly_" files as well so that the "Days Till Delete" = daysFromHidingToDeleting = 1.
- to summarize, the new rules are:
fileNamePrefix|daysFromUploadingToHiding|daysFromHidingToDeleting daily_|2|1| weekly_|30|1 monthly_364|1
- it looks like there's a logic error in the script preventing "weekly_" files from being created; it matches on "01", but it should match on "1"
# determine the prefix/interval of today's backup archive if "`date +"%m%d"`" == "0101" ; then # today is January 1st archivePrefix="yearly" elif `date +"%d"` == '01' ; then # today is the 1st of the month archivePrefix="monthly" elif `date +"%u"` == '01' ; then # today is the first day of the week = Monday archivePrefix="weekly" else # otherwise, we'll just call this a "daily" backup archivePrefix="daily" fi
- but note that the `date +"%u"` command does not have a leading 0
[maltfield@hetzner2 ~]$ date +"%u" 6 [maltfield@hetzner2 ~]$
- I changed the match to " == '1'". I'll verify that this is working in a month or so
- ..
- began investigating the repermission campaign solution. it looks like we need the "invite" pluigin https://discuss.phplist.org/t/gdpr-re-permission-campaigns-invite-plugin/4086
- I confirmed that we already have the invite plugin installed (looks like it came by default), but it's not enabled https://phplist.opensourceecology.org/lists/admin/?page=plugins
- here's the invite plugin page on phplist.com https://resources.phplist.com/plugin/invite
- and here's the source code (with README) for the invite plugin on github https://github.com/phpList/phplist-plugin-invite
- based on what I read above, I think this will be our procedure:
- create a new list
- import our subscribers from google spreadsheets into phplist, checking the box to *not* send them an email asking for them to confirm the subscription
- send out a new campaign (ideally as an actual OSEmail with content per Marcin's request) to all the subscribers, making sure to check the "invite" radio button on the Format tab. At the top of the campaign's [CONTENT], we should state that this is our first OSEmail from our phplist install, and that users will not recieve future updates unless they click the link (href=[CONFIRMATIONURL]).
- I have to confirm that, at some point in this process, the user is required to tick a box that they agree with our Privacy Policy (linking to the Privacy Policy on our wiki)
- the phplist manual specifically calls out repermission campaigns for gpdr https://www.phplist.org/manual/ch048_gdpr.xhtml
- it links to this article https://econsultancy.com/gdpr-examples-repermissioning-emails-campaigns/
Fri Nov 23, 2018
- I updated our phplist email template with actual social media links for twitter & facebook. rather than do a 'share this newsletter' approach, I just made it a generic "follow" or "like"
- twitter link is always now https://twitter.com/intent/user?screen_name=osecology
- facebook apparently doesn't have an equivalent without including scripts from their site. fuck that. the link is always now just pointing to the facebook page with no action https://www.facebook.com/OpenSourceEcology/
- I also removed the "web version" link
- I updated the template as stored on the wiki https://wiki.opensourceecology.org/wiki/File:OSEmail_phplist.tar.gz
- worked on a change to the phplist codebase per the "hack attempt" issues discussed in this thread https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-subscribelib2-php/4578/11
- submitted a PR https://github.com/phpList/phplist3/pull/444
- updated our privacy policy to include newsletters https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy#Newsletters
- I made some further improvements to the ajax form
<html><body>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>
<script type="text/javascript">
function checkform() {
// first, clear the response div from the previous attempts results
jQuery("#result").empty();
for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
if (document.phplistSubscribeForm.elements[fieldstocheck[i]].checked) {
} else {
jQuery("#result").empty();
alert("The following field is required: "+fieldnames[i]);
eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
} else {
if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
}
}
for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
return false;
}
}
if (! checkEmail()) {
alert("Email address is not valid");
return false;
}
return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name) {
fieldstocheck[fieldstocheck.length] = value;
fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name) {
groupstocheck[groupstocheck.length] = value;
groupnames[groupnames.length] = name;
}
function compareEmail() {
return (document.phplistSubscribeForm.elements["email"].value == document.phplistSubscribeForm.elements["emailconfirm"].value);
}
function checkEmail() {
var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.phplistSubscribeForm.elements["email"].value);
}
function checkGroup(name,value) {
option = -1;
for (i=0;i<document.phplistSubscribeForm.elements[name].length;i++) {
if (document.phplistSubscribeForm.elements[name][i].checked) {
option = i;
}
}
if (option == -1) {
alert ("Please enter your "+value);
return false;
}
return true;
}
function submitForm() {
// first, clear the response div from the previous attempts results
jQuery("#result").empty();
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#phplistSubscribeForm').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
// defines a function that's called when our ajax call suceeds (ie: gets a
// 200 response back)
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
// defines a function that's called when our ajax call fails (ie: gets a 500
// response back)
error: function (request, status, error) {
jQuery("#result").empty();
alert('Sorry, we were unable to process your subscription.');
}
});
}
</script>
<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">
<p>Signup for our newsletter!</p>
<!-- this simple form intentionally only requires the bare necessities:
[1] email address
[2] accept Privacy Policy
-->
<form method="post" action="" name="phplistSubscribeForm" id="phplistSubscribeForm">
<!-- [1] email address -->
<input type=text name=email required="required" placeholder="Email" size="30" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
<br/>
<!-- [2] accept Privacy Policy -->
<input type="checkbox" name="attribute4" value="on" class="attributeinput" id="attribute4" />
<label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a></label>
<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE Privacy Policy");</script>
<br/>
<!-- other strange hidden inputs per phplist's ajax example thread
* https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
-->
<input type="hidden" name="list[2]" value="signup" />
<input type="hidden" name="listname[2]" value="newsletter"/>
<input type="hidden" name="htmlemail" value="1" />
<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>
</form>
<!-- do some input sanity checking with js, then ajax() submit using jquery -->
<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>
</div>
<!-- this input will fill-in with results from the phplist server after the ajax
submission via jquery -->
<div id="result"></div>
<script type="text/javascript">
// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';
</script>
</body></html>
- fixed an issue where sending a new campaign redirected to the site on port 4443 (broken) by commenting-out the line defining the HTTP_HOST with the 4443 port
- fixed a modsecurity false-positive
- 981317 sqli
- sent a test campaign
- found an issue that really long words result in an overflow not word-wrapping, causing the header ose logo image from moving to the right.
Thr Nov 22, 2018
- Duncan responded to my thread about the "hack attempt" issue. He found a very obsecure config issue: there was no lists checked in the "lists to offer" section of the Subscribe Page. I suggested a code change to emit an error before the "exit" to make this issue easier to pinpoint https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-subscribelib2-php/4578/9
- I made some changes to the phplist ajax form, adding some css styles and html comments. I should still probably set the encoding and run it through a w3c checker to ensure compliance, but here's the current state of it
<html><body>
<style type="text/css">
#phplistSubscribeForm {
background-color: #f3f3f3;
border: solid 1px #a1a1a1;
padding: 10px;
//width: 600px;
}
#phplistSubscribeForm label, #phplistSubscribeForm input {
display: block;
width: 300px;
float: left;
margin-bottom: 10px;
}
#phplistSubscribeForm label {
text-align: right;
padding-right: 20px;
}
br {
clear: left;
}
</style>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>
<script type="text/javascript">
function checkform() {
// first, clear the response div from the previous attempts results
jQuery("#result").empty();
for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
if (document.phplistSubscribeForm.elements[fieldstocheck[i]].checked) {
} else {
jQuery("#result").empty();
alert("The following field is required: "+fieldnames[i]);
eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
} else {
if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
}
}
for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
return false;
}
}
if (! checkEmail()) {
alert("Email address is not valid");
return false;
}
return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name) {
fieldstocheck[fieldstocheck.length] = value;
fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name) {
groupstocheck[groupstocheck.length] = value;
groupnames[groupnames.length] = name;
}
function compareEmail() {
return (document.phplistSubscribeForm.elements["email"].value == document.phplistSubscribeForm.elements["emailconfirm"].value);
}
function checkEmail() {
var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.phplistSubscribeForm.elements["email"].value);
}
function checkGroup(name,value) {
option = -1;
for (i=0;i<document.phplistSubscribeForm.elements[name].length;i++) {
if (document.phplistSubscribeForm.elements[name][i].checked) {
option = i;
}
}
if (option == -1) {
alert ("Please enter your "+value);
return false;
}
return true;
}
function submitForm() {
// first, clear the response div from the previous attempts results
jQuery("#result").empty();
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#phplistSubscribeForm').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
// defines a function that's called when our ajax call suceeds (ie: gets a
// 200 response back)
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
// defines a function that's called when our ajax call fails (ie: gets a 500
// response back)
error: function (request, status, error) {
jQuery("#result").empty();
alert('Sorry, we were unable to process your subscription.');
}
});
}
</script>
<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">
<!-- this simple form intentionally only requires the bare necessities:
[1] email address
[2] accept Privacy Policy
-->
<form method="post" action="" name="phplistSubscribeForm" id="phplistSubscribeForm">
<!-- [1] email address -->
<label for="email">Email address *</label>
<input type=text name=email required="required" placeholder="" size="40" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
<br/>
<!-- [2] accept Privacy Policy -->
<label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label>
<input type="checkbox" name="attribute4" value="on" class="attributeinput" id="attribute4" />
<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script>
<br/>
<!-- other strange hidden inputs per phplist's ajax example thread
* https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
-->
<input type="hidden" name="list[2]" value="signup" />
<input type="hidden" name="listname[2]" value="newsletter"/>
<input type="hidden" name="htmlemail" value="1" />
<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>
<br/>
</form>
<!-- do some input sanity checking with js, then ajax() submit using jquery -->
<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>
</div>
<!-- this input will fill-in with results from the phplist server after the ajax
submission via jquery -->
<div id="result"></div>
<script type="text/javascript">
// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';
</script>
</body></html>
Sun Nov 07, 2018
- Duncan Cameron noted that in their environment, my change to throw an exception didn't change the exit code to 200. I looked it up, and found that if display_errors is enabled, then it will stay at 200--otherwise it's 500. https://github.com/phpList/phplist3/pull/432
- I replied to this thread https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-subscribelib2-php/4578
Sun Nov 04, 2018
- the phplist team approved & merged my PR to throw an exception when the ajax subscribe fails on the server, which changes the https status code from 200 ok to 500 internal server error https://github.com/phpList/phplist3/pull/432
- continuing with why the subscribelib2.php script thinks our ajax subscribe is a "hack attempt" and exits prematurely..
- it looks like it's because the the key in the key/value pair from the "list[2]=subscribe" argument is empty.
error_log( "made it to AC" );
if (isset($_POST['list']) && is_array($_POST['list'])) {
foreach ($_POST['list'] as $key => $val) {
error_log( "\tlist key|$key| val|$val|" ); if ($val == 'signup') {
$key = sprintf('%d', $key);
if (!empty($key)) {
error_log( "made it to AAA" );
$result = Sql_query(sprintf('replace into %s (userid,listid,entered) values(%d,%d,now())',
$GLOBALS['tables']['listuser'], $userid, $key));
$lists .= "\n * ".listname($key);
$subscriptions[] = $key;
addSubscriberStatistics('subscribe', 1, $key);
} else {
//# hack attempt...
error_log( "made it to AAB" );
exit;
}
}
}
}
error_log( "made it to AD" );
- the above code yields the following in the error log, indicating that the loop itreates through two distinct elements in the $_POST['list'] array. One's key is '2' and the other is empty.
[Sun Nov 04 18:56:57.061936 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AC, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.061951 2018] [:error] [pid 14919] [client 127.0.0.1:44604] \tlist key|2| val|signup|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.061977 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AAA, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.069666 2018] [:error] [pid 14919] [client 127.0.0.1:44604] \tlist key|| val|signup|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.069708 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AAB, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
- I created a thread on the phplist discourse forms asking why that "hack attempt" exit exists, and describing how it's breaking our ajax subscription flow via jquery.ajax(). I asked if I can remove it in a PR if it's not necessary https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-libsubscribe2-php/4578
- upon further research, I found that my ajax form is indeed submitting only "list[2]=signup" and that the "list[]=signup" paramater is actually being added by phplist (!) here https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/index.php#L247-L249
- I'm not sure what the $GLOBALS['pagedata']['lists'] data *should* have. In any case, I updated the discourse thread linked above
- in the meantime, I commented-out the "hack attempt" comment and encapsulating else statement
- now the subscribelib2.php code is ending at the "return 2" line here https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/admin/subscribelib2.php#L407
- ok, so now when I send the right inputs, the result div gets populated with the "thanks, you have been added to the newsletter..." message as returned by the phplist server's response. Horray! I also confirmed that when I submit with bad data, it fails properly (with a pop-up saying "Sorry, we were unable to process your subscription."
- it doesn't actually clear the success message after the fail message comes back, so that needs to be fixed next week
- ...
- I did some further digging into the code that checks for required attributes, regarding this bug I submitted earlier this week https://mantis.phplist.org/view.php?id=19503
- it appears that the first part of the if statement checks $subscribeagedata rather than checking the DB. I'm not sure what the advantage of this is, except for maybe making some attributes optional on some subscribe pages. The issue in my case was that my attribute in the relevant subscribe page (as determined by the GET variable (id=2) had the ToS attribute's "Is this attribute required" checkbox ticked, but then the "Check this box to use this attribute in the page" checkbox was *not* ticked. https://phplist.opensourceecology.org:4443/lists/admin/?page=spageedit&tk=2d2c23511c2b9762c09026d77c25cd31&id=2
- when I _did_ check the "use this attribute" checkbox (in addition to the "required" checkbox, which was already checked), the $subscribepagedata finally included the ToS attribute4 in the list of required attributes
[Sun Nov 04 19:41:22.044846 2018] [:error] [pid 30671] [client 127.0.0.1:58448] subscribepagedata:|Array\n(\n [ajax_subscribeconfirmation] => <h3>Thanks, you have been added to our newsletter</h3><p>You will receive an email to confirm your subscription. Please click the link in the email to confirm</p>\n [attribute004] => 4######0###1\n [attributes] => 4+\n [button] => Subscribe to the selected newsletters\n [emaildoubleentry] => no\n [footer] => </div>\r\n</div>\r\n</div><!-- ENDOF #mainContent-->\r\n</div><!-- ENDOF .wrapper -->\r\n</div><!-- ENDOF #container -->\r\n\r\n<div id="footer">\r\n<div id="footerframe">\r\n<ul class="list-unstyled">\r\n<li> </li>\r\n</ul>\r\n</div>\r\n</div>\r\n<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>\r\n<script type="text/javascript" src="admin/js/phplistapp.js"></script>\r\n<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>\r\n<script>\r\n/* do not remove this from here */\r\n$(document).ready(function(){\r\n if ( $('body').hasClass('invisible') ){\r\n myfunction();\r\n }\r\n\t$('#edit_list_categories input.form-control').tagsinput('refresh');\r\n\r\n});\r\n</script>\r\n</body>\r\n\r\n</html>\r\n\n [header] => <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />\r\n<meta name="theme-color" content="#2C2C2C"/>\r\n<link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" />\r\n<link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" />\r\n<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />\r\n</head>\r\n\r\n<body class="fixed invisible">\r\n\r\n<div id="container">\r\n\r\n<div id="header" class="navbar navbar-inverse">\r\n<div id="rack-functions" class="container">\r\n\t<h1 id="logo"><a href="http://phplist.opensourceecology.org" title="Visit our website">Open Source Ecology</a></h1>\r\n</div>\r\n</div>\r\n<div id="wrapper" class="container">\r\n<div id="mainContent">\r\n\r\n<div class="panel">\r\n<br />\r\n<div class="content well">\n [htmlchoice] => htmlonly\n [intro] => <h3>Subscribe to our newsletters</h3>\n [language_file] => \n [thankyoupage] => <h3>Thank you for subscribing to our newsletters.</h3>\r\nYour email address has been added to our system. You will receive a message with a request to confirm your membership. Please make sure to click the link in that message to confirm your subscription.\n [title] => poc\n [lists] => \n [rssdefault] => \n [rssintro] => \n [rss] => \n)\n|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
- I updated the bug report with my UX change suggestion
Fri Nov 02, 2018
- when I was poking at the ajax form & server-side code, I found that the ajax's "error" function was successfully called when I had a php-side syntax error, causing a 500 (internal server error) to be returned. So that's proven to work. It still does not, however, catch the FAIL response, which is included in a 200 response. I supose I could either fix the php code to return a non-200 (and submit a PR) or just catch FAIL messages from the ajax success() function and then call the ajax error() function from within success().
- I decided that the best way to handle this is for the php script to throw an exception instead of just print "FAIL", which causes a 500 status to be returned & is caught by the AJAX code
- I created a bug report for this here https://mantis.phplist.org/view.php?id=19524
- ugh, I discovered that phplist created their own sqllog() function in mysqli.inc, and then hard-codes log files to be written to "/tmp/". I wrote a post to the phplist Discourse forum askint the developers why they avoid php's built-in error_log() function https://discuss.phplist.org/t/why-doesnt-phplist-use-error-log/4570
- Xheni responded stating that they'd rather not spam the error log, and instead create a new log file in a new dir and lock it down with php. I stated that I don't think that's really possible, and that using trigger_error() would allow you to set messages as E_STRICT--which actually would not be written to the log file by default (only done so if the logging verbosity was increased on, say, a dev server). I also mentioned that they should use the php function sys_get_temp_dir() to get php's upload_tmp_dir rather than expect to be able to access '/tmp' as it's hardcoded now--since open_basedir may deny access to '/tmp/'.
- I reiterated that using trigger_error() would be able to send the messages to a log file that was readable only to root, something that I don't think would be possible to achieve within a php script.
- I finally found out what the form input VerificationCodeX is for. Apparently we just create it as a hidden field and do nothing with it. I gues the hope is that spam bot will attempt to fill it in. If it's anything but empty, we reject it as spam. I do something similar with an input field on email.michaelaltfield.net https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/admin/subscribelib2.php#L92-L119
- for some reason when subscfribelib2.php encounters an error, it returns lots of info, including the sucess messages. But when it successfully adds my user, it returns a generic payload without the success message.
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" > <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr"> <head> <meta http-equiv="pragma" content="no-cache" /> <meta http-equiv="Cache-Control" content="no-cache, must-revalidate" /> <meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" /> <meta name="Author" content="Michiel Dethmers - http://www.phplist.com" /> <meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" /> <meta name="Powered-By" content="phpList version 3.3.3" /> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <link rel="SHORTCUT ICON" href="./images/phplist.ico" />
- it appears that subscribelib2.php ends early (exit? return?). I should find out why. The goal here is to get phplist to return a 200 with the phplist configured success message on success. And it should return a 500 on failure, hopefully with some message as to why (though I'd accept a missing explination at this point)
- looks like I'm being detected as a hacker? I found that It's exiting just after the line "hack attempt" further research is required. https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/admin/subscribelib2.php#L255-L256
if (isset($_POST['list']) && is_array($_POST['list'])) {
foreach ($_POST['list'] as $key => $val) {
if ($val == 'signup') {
$key = sprintf('%d', $key);
if (!empty($key)) {
error_log( "made it to AAA" );
$result = Sql_query(sprintf('replace into %s (userid,listid,entered) values(%d,%d,now())',
$GLOBALS['tables']['listuser'], $userid, $key));
$lists .= "\n * ".listname($key);
$subscriptions[] = $key;
addSubscriberStatistics('subscribe', 1, $key);
} else {
//# hack attempt...
error_log( "made it to AAB" );
exit;
}
}
}
} error_log( "made it to AD" );
Tue Oct 30, 2018
- continuing with my quest to get the ajax form to work properly..
- I thought maybe the root cause of the empty error was the js console's message about CORS issues
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at ‘https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2’. (Reason: Credential is not supported if the CORS header ‘Access-Control-Allow-Origin’ is ‘*’).
</b>
# I changed the phplist config file from allowing '*' to just our domain
<pre>
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );
- but then the error changed to one about the Credentials issue
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2. (Reason: expected ‘true’ in CORS header ‘Access-Control-Allow-Credentials’).
- I did some research on this specific error, and it appears to pop-up when both the server and the client don't explicitly permit the Credentials flag, which would allow the server to set cookies across domains, creating possible CSRF vulnerabilities https://stackoverflow.com/questions/24687313/what-exactly-does-the-access-control-allow-credentials-header-do
- I checked the debugger for cookies being stored, and--sure enough--there was one named 'browsetrail'. I searched the entire phplist vhost dir for this file, and found a few releavant files. Namely, I found one of the js files we were including at the top of our script = phplistapp.js
[root@hetzner2 phplist.opensourceecology.org]# grep -irl 'browsetrail' * public_html/lists/admin/connect.php public_html/lists/admin/ui/dressprow/js/all.js public_html/lists/admin/ui/dressprow/js/all.min.js public_html/lists/admin/ui/default/js/all.js public_html/lists/admin/ui/default/js/all.min.js public_html/lists/admin/js/phplistapp.js [root@hetzner2 phplist.opensourceecology.org]#
- so I commented-out the include of this in our script, and the CORS erros went away entirely
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<!-- trying to prevent the 'browsetrail' cookie causing CORS errors in Access-Control-Allow-Credentials
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
-->
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
- After some digging, I found strikingly little logic behind the ajax backend that's listening to our form's requests. It's just this https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/index.php#L242-L272
case 'asubscribe': //# subscribe with Ajax
$_POST['subscribe'] = 1;
if (isset($_GET['email']) && !isset($_POST['email'])) {
$_POST['email'] = $_GET['email'];
}
foreach (explode(',', $GLOBALS['pagedata']['lists']) as $listid) {
$_POST['list'][$listid] = 'signup';
}
$_POST['htmlemail'] = 1; //# @@ should actually be taken from the subscribe page data
$success = require 'admin/subscribelib2.php';
$result = ob_get_contents();
ob_end_clean();
if (stripos($result, $GLOBALS['strEmailConfirmation']) !== false ||
stripos($result, $pagedata['thankyoupage']) !== false
) {
if (!empty($pagedata['ajax_subscribeconfirmation'])) {
$confirmation = $pagedata['ajax_subscribeconfirmation'];
} else {
$confirmation = getConfig('ajax_subscribeconfirmation');
}
if (empty($confirmation)) {
echo 'OK';
} else {
echo $confirmation;
}
exit;
} else {
echo 'FAIL';
}
break;
- so basically all I have to work with is if it responded with 'FAIL' or 'OK'. In both cases, the jquery ajax function will hook the "success" function.
- this is very bad as we're basically depending on our js form for proving that the user accepted the Privacy Policy. Indeed, if I remove the JS on the form that ensures the "I agree..." checkbox is ticked and submit the form with it unticked, then phplist happily subscribes the user! This is *wrong*!
- the best solution, then, is to update this code
- unfortunaetly, the ajax solution that we're updating is being deprecated in the next release of phplist4 in favor of the REST API
- the attribute that I need to verify identifies itself as "attribute4" — which is hardly specific
- note, however, that the attribute does have the property (meta attribute?) "is this attribute required"
- therefore, probably the best way to write this robustly is to get a list of all the attributes that are requried & ensure that they are present. If any required attributes are not present (or have 0 /blank/ null values), then do *not* subscribe the user and return an error
- so, actually, the block above does require another script--which actually does the logic for adding the user. Indeed, there is some logic in there for checking for required fields. https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/admin/subscribelib2.php#L18=L81
//# Check if input is complete
$allthere = 1;
$subscribepagedata = PageData($id);
if (isset($subscribepagedata['language_file']) && is_file(dirname(FILE).'/../texts/'.basename($subscribepagedata['language_file']))) {
@include_once dirname(FILE).'/../texts/'.basename($subscribepagedata['language_file']);
}
// Allow customisation per installation
if (is_file($_SERVER['DOCUMENT_ROOT'].'/'.basename($GLOBALS['language_module']))) {
include_once $_SERVER['DOCUMENT_ROOT'].'/'.basename($GLOBALS['language_module']);
}
if (!empty($data['language_file']) && is_file($_SERVER['DOCUMENT_ROOT'].'/'.basename($data['language_file']))) {
include_once $_SERVER['DOCUMENT_ROOT'].'/'.basename($data['language_file']);
}
$required = array(); // id's of missing attribbutes '
if (count($subscribepagedata)) {
$attributes = explode('+', $subscribepagedata['attributes']);
foreach ($attributes as $attribute) {
if (isset($subscribepagedata[sprintf('attribute%03d',
$attribute)]) && $subscribepagedata[sprintf('attribute%03d', $attribute)]
) {
list($dummy, $dummy2, $dummy3, $req) = explode('###',
$subscribepagedata[sprintf('attribute%03d', $attribute)]);
if ($req) {
array_push($required, $attribute);
}
}
}
} else {
$req = Sql_Query(sprintf('select * from %s', $GLOBALS['tables']['attribute']));
while ($row = Sql_Fetch_Array($req)) {
if ($row['required']) {
array_push($required, $row['id']);
}
}
}
if (count($required)) {
$required_ids = implode(',', $required);
// check if all required attributes have been entered;
if ($required_ids) {
$res = Sql_Query("select * from {$GLOBALS['tables']['attribute']} where id in ($required_ids)");
$allthere = 1;
$missing = '';
while ($row = Sql_Fetch_Array($res)) {
$fieldname = 'attribute'.$row['id'];
$thisonemissing = 0;
if ($row['type'] != 'hidden') {
$thisonemissing = empty($_POST[$fieldname]);
if ($thisonemissing) {
$missing .= $row['name'].', ';
}
$allthere = $allthere && !$thisonemissing;
}
}
$missing = substr($missing, 0, -2);
if ($allthere) {
$missing = '';
}
}
} else {
$missing = '';
}
- so the "$requried" var should store a list of all the arguments that are required. I added some test debug code to output this array, and it only contained one element = "1", which I'm assuming is the email address. further research is required
- also, I realized that there doesn't appear to be a per-list attribute config section, which makes me think that you can't have distinct attribute sets for different subscriber lists, which may be a problem. Further research into this is required
- ...
- I dug into the db further to see what I could dig up: I found only 1 row in the table 'phplist_user_attribute' with my 'termsofservice' attribute, which is indeed marked as required
MariaDB [phplist_db]> describe phplist_user_attribute; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | name | varchar(255) | NO | MUL | NULL | | | type | varchar(30) | YES | | NULL | | | listorder | int(11) | YES | | NULL | | | default_value | varchar(255) | YES | | NULL | | | required | tinyint(4) | YES | | NULL | | | tablename | varchar(255) | YES | | NULL | | +---------------+--------------+------+-----+---------+----------------+ 7 rows in set (0.00 sec) MariaDB [phplist_db]> select * from phplist_user_attribute; +----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+ | id | name | type | listorder | default_value | required | tablename | +----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+ | 4 | I agree to the OSE <a href=\'https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy\'>Privacy Policy</a>. | checkbox | 0 | | 1 | termsofservice | +----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+ 1 row in set (0.00 sec) MariaDB [phplist_db]>
- note there were two other tables with 'attribute' in their name, which appear to track & store the fact that each user has accepted the terms of service as required by gpdr
MariaDB [phplist_db]> select * from phplist_admin_attribute -> ; Empty set (0.00 sec) MariaDB [phplist_db]> select * from phplist_admin_attribute; Empty set (0.00 sec) MariaDB [phplist_db]> select * from phplist_user_user_attribute; +-------------+--------+-------+ | attributeid | userid | value | +-------------+--------+-------+ | 4 | 3 | on | | 4 | 6 | on | | 4 | 7 | on | | 4 | 8 | on | +-------------+--------+-------+ 4 rows in set (0.00 sec) MariaDB [phplist_db]>
- I submitted a bug report with all these details to the phplist mantis https://mantis.phplist.org/view.php?id=19503
- further testing should be done, modifying the script to ensure that it always populates the $required attributes array and checks to see if theyr'e missing
Mon Oct 29, 2018
- revisiting my ajax solution for phplist, I have 3 outstanding issues
- I need to harden the ACCESS_CONTROL_ALLOW_ORIGIN options and figure out how to set it up for mulitple domains
- I need to format it better since I removed the tables
- I need to get it to handle errors properly
- the last one, error catching, was discussed here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491
- the full code of the ajax form for error handling is given here https://github.com/scotch-io/ajax-forms-jquery/blob/master/magic.js
- an explination is found here https://scotch.io/tutorials/submitting-ajax-forms-with-jquery
- I tried to get some specific info, but all I got was a status code of 0 and a status text of 'error' — both when I give it proper & improper data.
Fri Oct 26, 2018
- I made some progress on the ajax form. did some cleanup & better functionality when JS is disabled
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html 100% 5009 49.5KB/s 00:00
user@ose:~$ cat phplistAjax.html
<html><body>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>
<script type="text/javascript">
function checkform()
{
for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
if (document.subscribeform.elements[fieldstocheck[i]].checked) {
} else {
alert("The following field is required: "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
} else {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
}
}
for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
return false;
}
}
if (! checkEmail()) {
alert("Email address is not valid");
return false;
}
return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
fieldstocheck[fieldstocheck.length] = value;
fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
groupstocheck[groupstocheck.length] = value;
groupnames[groupnames.length] = name;
}
function compareEmail()
{
return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}
function checkEmail()
{
var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.subscribeform.elements["email"].value);
}
function checkGroup(name,value)
{
option = -1;
for (i=0;i<document.subscribeform.elements[name].length;i++) {
if (document.subscribeform.elements[name][i].checked) {
option = i;
}
}
if (option == -1) {
alert ("Please enter your "+value);
return false;
}
return true;
}
</script>
<script type="text/javascript">
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
async: true,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
//contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
</script>
<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">
<form method="post" action="" name="subscribeform" id="subscribeform">
<div class="required"><label for="email">Email address *</label></div>
<input type=text name=email required="required" placeholder="" size="40" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
<input type="hidden" name="htmlemail" value="1" />
<input type="checkbox" name="attribute4" value="on" class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span>
<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script>
<input type="hidden" name="list[2]" value="signup" />
<input type="hidden" name="listname[2]" value="newsletter"/>
<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>
</form>
<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>
</div>
<script type="text/javascript">
// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';
</script>
</body></html>
user@ose:~$
- meeting with Marcin about phplist
- I explained GDPR and how Alex's site is not properly meeting our obligations
- I asked how we should integrate our phplist newsletter to social media, and Marcin said don't worry about it; we should just delete the links to like/retweet the newsletter posting for now. Perhaps in the future we can script a tool to automatically send a campaign when a new post is made to our wordpress site, then post to social media & add-back the "like" and "retweet" links then. But, for now, we just need to get him suited to send newsletters without jumping through 4 distinct gmail accounts.
- I asked about attributes. He said that we should import all existing lists' attributes into phplist. therefore, we're going to want to enumerate all of this info into our gdpr-compliant Privacy Policy
- I explained how phplist ajax & rest api works (or doesn't work) to add users to our subscription list without requring them to leave out other domain. I'm currently ironing-out final issues in the ajax solution.
- I explained how the repermission campaign would be requried after imports.
- we discussed how we would have multiple lists one for each of the existing lists, and then we would create one more list for the general "ose newsletter" which only needs the 3 basic & required attributes = timestamp, email address, & accepted PP.
Wed Oct 24, 2018
- continuing with my efforts to get a custom form to add a subscriber to our phplist site using AJAX..
- the source of offending modsecurity rule id=960010 is in /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
[root@hetzner2 ~]# grep '960010' -B 30 -A 2 /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
# Restrict which content-types we accept.
#
# TODO Most applications support only two types for request bodies
# because that is all browsers know how to produce. If you are using
# automated tools to talk to the application you may be using other
# content types and would want to change the list of supported types.
#
# Note though that ModSecurity parses only three content types:
# application/x-www-form-urlencoded, multipart/form-data request and
# text/xml. The protection provided for any other type is inferior.
#
# TODO There are many applications that are not using multipart/form-data
# types (typically only used for file uploads). This content type
# can be disabled if not used.
#
# NOTE We allow any content type to be specified with GET or HEAD
# because some tools incorrectly supply content type information
# even when the body is not present. There is a rule further in
# the file to prevent GET and HEAD requests to have bodies to we're
# safe in that respect.
#
# NOTE Use of WebDAV requires "text/xml" content type.
#
# NOTE Philippe Bourcier (pbourcier AT citali DOT com) reports
# applications running on the PocketPC and AvantGo platforms use
# non-standard content types:
#
# M-Business iAnywhere application/x-mal-client-data
# UltraLite iAnywhere application/octet-stream
#
SecRule REQUEST_METHOD "!^(?:GET|HEAD|PROPFIND|OPTIONS)$" "phase:1,chain,t:none,block,msg:'Request content type is not allowed by policy',rev:'2',ver:'OWASP_CRS/2.2.9',maturity:'9',accuracy:'9',id:'960010',tag:'OWASP_CRS/POLICY/ENCODING_NOT_ALLOWED',tag:'WASCTC/WASC-20',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/EE2',tag:'PCI/12.1',severity:'2',logdata:'%{matched_var}'"
SecRule REQUEST_HEADERS:Content-Type "^([^;\s]+)" "chain,capture"
SecRule TX:0 "!^%{tx.allowed_request_content_type}$" "t:none,ctl:forceRequestBodyVariable=On,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-OWASP_CRS/POLICY/CONTENT_TYPE_NOT_ALLOWED-%{matched_var_name}=%{matched_var}"
[root@hetzner2 ~]#
- now, I am setting the enctype of the form to "multipart/form-data" in the ajax form, but modsecurity is complaining that it's "text/plain"
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
- I found a list of supported data types in /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
[root@hetzner2 ~]# grep 'application/xml' /etc/httpd/modsecurity.d/* grep: /etc/httpd/modsecurity.d/activated_rules: Is a directory /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf: setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json', \ [root@hetzner2 ~]#
- I also found a note that the offending data type (text/plain) was indeed added to the acceptable list in the CRS 3.0 back in 2016; not sure why we don't have that https://github.com/SpiderLabs/owasp-modsecurity-crs/issues/208
- so I could update the CRS's config to include "text/plain", but I think the best solution is to fix the AJAX code to send the correct data type.
- the offending code is probably somewhere in this jquery.ajax() call (as obtained from the example zip in the phplist forum thread https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',
dataType: 'html',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
- I've never used jquery, but my guess is that we need to set the "dataType" here; it's documented here https://api.jquery.com/jQuery.ajax/
- oh, hmm, actualy reading the docs suggest that the 'dataType' defines what data type we expect _back_ from the server https://api.jquery.com/jQuery.ajax/
The type of data that you're expecting back from the server. If none is specified, jQuery will try to infer it based on the MIME type of the response (an XML MIME type will yield XML, in 1.4 JSON will yield a JavaScript object, in 1.4 script will execute the script, and anything else will be returned as a string). The available types (and the result passed as the first argument to your success callback) are:
- so I guess I need to dig into the jquery.serialize() command
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
...
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
...
}
- so the documentation on the serialize function explicitly states that it creates a string https://api.jquery.com/serialize/
- ah, so the option I want is "contentType" https://api.jquery.com/jQuery.ajax/
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize(); jQuery.ajax( {
type: 'POST',
data: data,
url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',
dataType: 'html',
contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
}, error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
- there's a new modsecurity alert
[Wed Oct 24 20:43:47.876714 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Multipart parsing error (init): Multipart: Boundary not found in C-T. [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"] [Wed Oct 24 20:43:47.876765 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "REQBODY_ERROR" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "151"] [id "960912"] [rev "1"] [msg "Failed to parse request body."] [data "Multipart: Boundary not found in C-T."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]
- so this is probably because the content is--in fact--actually "text/plain" there's no multipart boundray since I just lied about what the mime type is. The solution should still be to get serialize() to output the proper mime type
- well, it appears that the alternative solution (as is often used for form data including binary content--such as images), is to use the FormData, but this is a browser-specific thing https://developer.mozilla.org/en-US/docs/Web/API/FormData
- so proceeding with FormData would likely limit the browsers in which this ajax form will work. It's really not requred for the small fields needed for subscriber attributes data in this form. so let's just ignore the modsecurity role
- well, I tried to overwrite the existing allowable mime types in our vhost file (making restore from backups easier). that didn't work
- I tried to just add the rule id = 960010 to the long list of SecRuleRemoveById list in the vhost file; that also didn't work
- the only way it worked as if I actually edited the list in the /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf file, adding "text/plain" to the list (followed by an apache reload) https://stackoverflow.com/questions/41604674/customizing-apache-mod-security-to-accept-content-type-text-plain
[root@hetzner2 modsecurity.d]# pwd /etc/httpd/modsecurity.d [root@hetzner2 modsecurity.d]# grep -C5 'text/plain' modsecurity_crs_10_config.conf SecAction \ "id:'900012', \ phase:1, \ t:none, \ setvar:'tx.allowed_methods=GET HEAD POST OPTIONS', \ setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json|text/plain', \ setvar:'tx.allowed_http_versions=HTTP/0.9 HTTP/1.0 HTTP/1.1', \ setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/', \ setvar:'tx.restricted_headers=/Proxy-Connection/ /Lock-Token/ /Content-Range/ /Translate/ /via/ /if/', \ nolog, \ pass" [root@hetzner2 modsecurity.d]#
- so this time it just got a 200 from the server (logs all look clean), but the user didn't get added. The js console shows some (unrelated?) errors about our jquery script file
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More] jquery.min.js:112:77 SyntaxError: missing ) after argument list[Learn More] phplistAjax.php:1:58 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/ jquery.min.js:127:343 TypeError: $(...).tagsinput is not a function[Learn More] phplistAjax.php:7:2 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2 ready https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
- I checked out the sourcecode of our phplist subscribe page, and I discovered that it has its own checkform() js function, and some helper functions. I just copied these to my ajax form https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
- unrealted, but I saw that at the top of this sourcecode is a php command that for some reason came out as text, not processed by php
<head> <meta http-equiv="pragma" content="no-cache" /> <meta http-equiv="Cache-Control" content="no-cache, must-revalidate" /> <meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" /> <meta name="Author" content="Michiel Dethmers - http://www.phplist.com" /> <meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" /> <meta name="Powered-By" content="phpList version 3.3.3" /> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <link rel="SHORTCUT ICON" href="./images/phplist.ico" /> <title>Subscribe to our Newsletters</title><meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" /> <meta name="theme-color" content="#2C2C2C"/> <link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" /> <link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" /> <link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" /> </head>
- it looks like this comes from pagetop_minified.php
[root@hetzner2 lists]# pwd
/var/www/html/phplist.opensourceecology.org/public_html/lists
[root@hetzner2 lists]# cat admin/ui/dressprow/pagetop_minified.php
<?php
/*
We request you retain the full headers below including the links.
This not only gives respect to the large amount of time given freely
by the developers, but also helps build interest, traffic and use of
phpList, which is beneficial to it's future development.
Michiel Dethmers, phpList Ltd 2003 - 2015
*/
?>
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="<?php echo $_SESSION['adminlanguage']['iso']?>" lang="<?php echo $_SESSION['adminlanguage']['iso']?>" dir="<?php echo $_SESSION['adminlanguage']['dir']?>">
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phplist version <?php echo VERSION?>" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" id="favicon" href="./images/phplist.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<link rel="stylesheet" href="ui/dressprow/css/all.min.css?v=<?php echo filemtime(dirname(FILE).'/css/all.min.css'); ?>" />
<?php
if (isset($GLOBALS['config']['head'])) {
foreach ($GLOBALS['config']['head'] as $sHtml) {
echo $sHtml;
echo "\n";
echo "\n";
}
}
[root@hetzner2 lists]#
- interstingly, there's php lines above it (ie: the "phplist version") execute without issues
- this is somewhat related, but not really https://mantis.phplist.org/view.php?id=18447
- ah, this is it https://mantis.phplist.org/view.php?id=19307
- looks like the fix was to just remove it https://github.com/phpList/phplist-ui-bootlist/pull/62
- anyway, it looks like the error "SyntaxError: missing ) after argument list" was caused by my attribute with a quote in it. Xheni warned me about this here https://mantis.phplist.org/view.php?id=19436
- I replaced my links' encapsulating double qoutes with single quotes as a temp fix
- ok, better fix, update the attribute to use single quotes in phplist's admin ui
- so actually the issue is quotes, not slashes.
- ok, that elminated the error entirely. Now I just have
Loading mixed (insecure) display content “http://powered.phplist.com/images/3.3.3/power-phplist.png” on a secure page[Learn More] jquery.min.js:112:77 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/ jquery.min.js:127:343 TypeError: $(...).tagsinput is not a function[Learn More]
- this is frustrating, but I confirmed that signup *does* work from our actual subscribe page https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1
- I noticed that there were a few js files included in the phplist site's code, so I added these to my ajax form
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script> <script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script> <script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
- now I get a pop-up alert = "Sorry, we were unable to process your subscription."
- someone else posted about this issue a couple years ago https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
- ah, another point. I have the URL wrong! per https://discuss.phplist.org/t/ajax-subscribe-api/974/15
url: 'https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1',
- but, in fact, it should use 'asubscribe' intead of 'subscribe' = https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=1
- per the thread above, I also changed it to default to html & not confirm email address
- it's still not working. this is taking for fucking ever. I gotta get some POC. I'll simplify it by stripping out the "I agree" checkbox for now. If I can fucking get that to work, I'll try to add a simple checkbox and work from there..
- I created a new list and cooresponding subscribe page that's as simple as possible: it's just an email address https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=2
- ugh, even that won't work.
- oh, I got it to work! I had to set both the name and the id attributes of the form field to "subscribeform". Note that the form coming from phplist's html only sets the 'name' attribute, but the jquery serialize() function is called against the 'id' attribute. otherwise, it probably just sent an empty string to the subscribe page!
<form method="post" action="" name="subscribeform" id="subscribeform">
- woohoo! I switched back to the list=1 (with the checkbox agreeing to our PP), and it worked! When the checkbox is *not* ticked, a JS pop-up yells at you:
The following field is required: I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.
- unfortunately, it still says that it fails even when it succeeds, as the thread above pointed out here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
- fwiw, this is what woked today. Tomorrow, I should look into making this visible only if JS is enabled. Otherwise, I should just display an anchor link the user to our phplist subscription page
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html 100% 5329 53.4KB/s 00:00
user@ose:~$ cat phplistAjax.html
<html><body>
<head>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<script type="text/javascript">
function checkform()
{
for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
if (document.subscribeform.elements[fieldstocheck[i]].checked) {
} else {
alert("The following field is required: "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
} else {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
}
}
for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
return false;
}
}
if (! checkEmail()) {
alert("Email address is not valid");
return false;
}
return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
fieldstocheck[fieldstocheck.length] = value;
fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
groupstocheck[groupstocheck.length] = value;
groupnames[groupnames.length] = name;
}
function compareEmail()
{
return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}
function checkEmail()
{
var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.subscribeform.elements["email"].value);
}
function checkGroup(name,value)
{
option = -1;
for (i=0;i<document.subscribeform.elements[name].length;i++) {
if (document.subscribeform.elements[name][i].checked) {
option = i;
}
}
if (option == -1) {
alert ("Please enter your "+value);
return false;
}
return true;
}
</script>
<script type="text/javascript">
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
async: true,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
//contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
</script>
</head>
<form method="post" action="" name="subscribeform" id="subscribeform"><table border=0>
<tr><td><div class="required"><label for="email">Email address *</label></div></td>
<td class="attributeinput"><input type=text name=email required="required" placeholder="" size="40" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script></td></tr><input type="hidden" name="htmlemail" value="1" />
<tr><td colspan="2">
<input type="checkbox" name="attribute4" value="on" class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span><script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script></td></tr>
</table><input type="hidden" name="list[2]" value="signup" /><input type="hidden" name="listname[2]" value="newsletter"/><div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div><input type=submit name="subscribe" value="Subscribe to the selected newsletters" onClick="return checkform();"> <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website" ><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0" /></a></p></div>
<button class='button'
onclick="if (checkform()) {submitForm();} return false;"
>Subscribe</button>
<div id="result" style="color: red;"></div>
</div>
</body></html>
user@ose:~$
Tue Oct 23, 2018
- Sam got back to me about integrating a signup list into wordpress via AJAX or the REST API. They said the API doesn't yet support attributes (that conflicts with the example I saw yesterday, though). Sam said adding that feature to the API will come with time, but we could fund it to be built in 6-8 weeks for ~$4,500. Of course, we could also just develop it ourselves and PR it into their repo (it uses Symfony and Doctrine) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/4
- Sam didn't give a direct answer, but suggested that it's probably possible to do with ajax
- fuck it; let's see if I can manually create a form that does what I want with ajax. I'll use this old example as a template https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
- I put the file up on the microfactory site in the wp-content dir.
- I got a success message (Thank you for your registration. Please check your email to confirm.), but it didn't add to the subscribers list. I checked the html console, and I got an error due to Cross-Origin
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/?p=subscribe&id=1. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘http://phplist.opensourceecology.org’).
- I found that phplist inclues logic to define the "Acces-Control-Allow-Origin" header based on a config file option
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'Allow-Origin' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'ACCESS_CONTROL_ALLOW_ORIGIN' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
public_html/lists/admin/init.php:if (!defined('ACCESS_CONTROL_ALLOW_ORIGIN')) {
public_html/lists/admin/init.php: define('ACCESS_CONTROL_ALLOW_ORIGIN', $GLOBALS['scheme'].'://'.$_SERVER['HTTP_HOST']);
[root@hetzner2 phplist.opensourceecology.org]#
- and here's the relevant option in the config file https://resources.phplist.com/system/config/access_control_allow_origin
- apparently the protcol (https://) must be specified. I finally got a new error after adding this line to the phplist config.php file
// allow AJAX queries to add subscribers to our db from other domains
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );
- now the error is
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More] jquery.min.js:112:77 SyntaxError: missing ) after argument list[Learn More] phplistAjax.php:1:58 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/ jquery.min.js:127:343 TypeError: $(...).tagsinput is not a function[Learn More] phplistAjax.php:7:2 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2 ready https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange
- this time jquery just fucking fetched the whole form and put it on the page somehow.
<div class="panel"> <br> <div class="content well"><h3>Subscribe to our newsletters</h3> <div class="error"><span class="required">* required fields are marked red</span></div> <div class="error missing">The following required values are missing: Email addresses you entered do not match</div><br> <form method="post" action="" name="subscribeform"><table border="0"> <tbody><tr><td><div class="required"><label for="email">Email address *</label></div></td> <td class="attributeinput"><input name="email" required="required" placeholder="testa@example.com" size="40" id="email" type="text"> </td></tr> <tr><td><div class="required"><label for="confirm">Confirm your email address *</label></div></td> <td class="attributeinput"><input name="emailconfirm" required="required" value="" size="40" id="confirm" type="text"> </td></tr><tr><td colspan="2"> <span class="attributeinput"> <input name="textemail" value="1" id="textemail" type="checkbox"></span> <span class="attributename"><label for="textemail">I prefer to receive emails in Text format</label></span> </td></tr> <tr><td colspan="2"> <input name="attribute4" value="on" class="attributeinput" id="attribute4" type="checkbox"> <span class="required"><label for="attribute4">I agree to the OSE <a href="https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy">Privacy Policy</a>. *</label></span></td></tr> </tbody></table><input name="list[2]" value="signup" type="hidden"><input name="listname[2]" value="newsletter" type="hidden"><div style="display:none"><input name="VerificationCodeX" value="" size="20" type="text"></div><input name="subscribe" value="Subscribe to the selected newsletters" onclick="return checkform();" type="submit"> <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website"><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0"></a></p></div> </div>
- I used this html form data to merge into the example one
- I hit a modsecurity issue with rule #960010
Request content type is not allowed by policy
- I think it may be related to the content type submitted by the form somewhere defaulting to "text/plain" *shrug*
- I had lost my password for our phplist install due to my ssd crash; I reset it (noting that you cannot login with your email address; you must use your username--as addressed in the password reset email. In my case, 'maltfield').
- I also added an admin for marcin
- I confirmed that the shared 'admin' user is in our shared ose keepass db
Mon Oct 22, 2018
- Marcin sent me our other 3 email spreadsheets; I dug through them to see what attributes we collected
- according to phplist plugin page, the rest api doesn't support setting attributes https://resources.phplist.com/plugin/restapi
- however, there does appear to be support for non-email attributes per this document, even though it does mention on the main page of this github repo that it's deprecated (the replacement for php4 isn't stable, however) https://github.com/phpList/phplist-plugin-restapi/blob/master/plugins/restapi2/docs/api.apib#L185-189
- the replacement api's repo does say that it can work for php3 https://github.com/phpList/rest-api
This new REST API can also be used to provide REST access to an existing phpList 3 installation. For this, the phpList 3 installation and the phpList 4 installation with the REST API need to share the same database. For security reasons, the REST APIs from phpList 3 and phpList 4 should not be used for the same database in parallel, though.
Sat Oct 20, 2018
- I did a search for wordpress plugins using the keyword "phplist". Here's some noteable results
- Sign Me Up https://wordpress.org/plugins/sign-me-up/
- - last updated 3 years ago
- + explicitly lists that it connects with phplist using AJAX
- - says that it doesn't support additional attributes (email address only). This will probably not work as we need to at least have a checkbox for the user to accept our gdpr-ready Privacy Policy
- there's a lot of info about this plugin here https://www.jaromy.net/wordpress-plugins/sign-me-up[[2]]
- WP PHPList https://wordpress.org/plugins/phplist-form-integration/
- - last updated 9 years ago
- yPHPlista https://wordpress.org/plugins/yphplista/
- - last updated 8 years ago https://wordpress.org/plugins/yphplista/
- Sign Me Up https://wordpress.org/plugins/sign-me-up/
- I tried searching the phplist forums for "wordpress" to dig through recent discussions about integrating phplist with wordpress
- I found this interesting discussion hinting to having a phplist plugin automatically send email campaigns after a new wordpress post is in. I really like this idea (ie: RSS scraping -> email campaign). Prior to sending the email campaign, we could post a link to it on facebook, twitter, etc. Then we could add a link to "like" or "retweet" the post (on social media) to the original post (on wordpress) within the email template. https://discuss.phplist.org/t/how-is-content-generated-in-wordpress/1656/6
- so 6 months ago someone asked my question: is there an official wp plugin for phplist? Sam Tuke (CEO of phplist) responded stating that there isn't but it's needed. And expounded on how the AJAX solution is the old way and that the future phplist wp plugin would work via the REST API (I like this much better too) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894
- this may work with the AJAX way https://discuss.phplist.org/t/wordpress-integration-newsletter-sign-up-plugin/1723/15
- And there was this old post for 2016 about the 9-year-old "WP PHPList" (phplist-form-integration) wordpress plugin https://discuss.phplist.org/t/phplist-wordpress-integration-solved/2279/2
- I lost my password for phplist's discourse site due to my ssd crash, so I generated a new one
- Interestingly, it looks like phplist's discourse site is SSO using wordpress as the account backend
- I replied to this post, asking Sam if there's an option for sending our new subscriber's attributes to the stable version phplist via either AJAX or REST https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/3
Fri Oct 19, 2018
- I'm back in NYC after restoring my laptop following my ssd crash
- sent Sara information about wp-cli and managing wordpress
- I also sent an email to Marcin asking if he authorizes them having ssh and/or shraed ose keepass access
- specifically, the plugin that Sara wanted to install was this https://wordpress.org/plugins/custom-css-js
- the above plugin is freemium. the reviews suggested this free alterantive for more features, but it's a less popular plugin https://wordpress.org/plugins/scripts-n-styles/
- Marcin sent me a confusing email about utilization of google analytics. One of our first conversations with Marcin back in 2017 was about how OSE was working to migrate _away_ from Google services. We've since stripped analytics from all our sites. We looked at Piwik, but--due to security issues--we decided to go with awstats.
- I restated that we use awstats, and asked what are the benefits of using google analytics
- I added the microfactory site to our awstast generation cron job, and confirmed that I could view all the stats so far this month https://awstats.opensourceecology.org:4443/awstats.microfactory.opensourceecology.org.html
- In general, I'm very confused about where the line is drawn between giving up on FLOSS and moving to some closed-source SaaS solution here at OSE.
- I've been hacking the hell out of phplist so that we can use an ugly FLOSS solution, but Mailchimp would be much better, and very cheap for our needs.
- We still don't have a decent issue tracker. Mantis is a long-standing TODO, but we could use Jira for free because we're a nonprofit.
- Google Analytics is surely more powerful thatn awstats, but does that really justify the use of closed-source tools that exploit our visitor's data? Why draw the line here between awstats & GA but not use Jira or Mailchimp like most other nonprofits?
- Marcin sent me an error when trying to post our email thread about installing a wordpress plugin to the wiki
- fixed by modsec whitelist
- 950019, generic attack = email injection
- fixed by modsec whitelist
- I updated the documentation on the wiki for the quick-start guide to configuring 2FA for our wordpress sites using the Google Authenticator plugin https://wiki.opensourceecology.org/wiki/2FA#Quick_Start_Guide
- I added a step to enable relaxed mode, so the user's phone only has to be correct within 4 minutes, rather than 30 seconds.
- I recommended use of the oandbackup app, which is the FLOSS replacement for Titanium Backup https://f-droid.org/en/packages/dk.jens.backup/
- ...
- began to revisit my install of the "Newsletter Sign-Up" wordpress plugin on the fef site
- I logged into the fef wordpress ite
- I navigated to the Plugins -> "Newsletter Sign-Up" -> "Settings" link (note that this redirects to the main panel nav bar's "Newsl. Sign-Up" link.
- I changed "Select your mailinglist provider" to "PHPList"
- our subscriber link is this, so I left the "PHPList list ID" at "1" https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
- I set the "Newsletter form action" to "https://phplist.opensourceecology.org/lists/?p=subscribe&id=1"
- this plugin appears to be mostly built around the trick of having someone signup when leaving a comment. That's a feature that we may want to disable. Mostly we want a widget and a form embedded into a post/page.
- I picked a bad site, fef, for testing widgets. it doesn't use them!
- I tested adding a form for signing up to the phplist newsletter to the lola page by adding "[nsu_form]" to the content of the post https://fef.opensourceecology.org/2015/09/03/lola/
- when I submitted on this form, it just fucking redirected me to the phplist site. it didn't even auto-fill the form! what rubbish! what we want here is for a seamless submittion to the phplist site without the user having to leave the site that they are on.
- ugh, I just realized that the "Newsletter Sign-Up" form's main plugin page on the wp site shows a message indicating that the plugin is no longer maintained (it was last updated 11 months ago) https://wordpress.org/plugins/newsletter-sign-up/
- I deactivated the "Newsletter Sign-Up" plugin and deleted the related content & widgets from the fef site.
- looks like I lost my logs from when I discovered the "Newsletter Sign-Up" and similar wordpress plugins
- I dug up some options
- https://wordpress.org/plugins/sign-me-up/
- - last updated 3 years ago
- https://wordpress.org/plugins/yphplista/
- - last updated 8 years ago
- https://wordpress.org/plugins/sign-me-up/
- it looks like phplist.com (not .org) offers a solution to this issue, the so-called "AJAX" solution so subscribres can signup "...and stay on your website" https://www.phplist.com/ajaxdemo
- found some info on enabling this the '.org' way. note that there may be issues with the ACCESS_CONTROL_ALLOW_ORIGIN https://discuss.phplist.org/t/ajax-subscribe-api/974/7
define('ACCESS_CONTROL_ALLOW_ORIGIN', 'www.mydomain.de');
define('PUBLIC_PROTOCOL','https');
- I found a few resources about ajax on phplist.com's wiki https://resources.phplist.com/start?do=search&id=ajax
- ugh, this just references the 2x no-longer-maintained plugins I found above https://resources.phplist.com/develop/wordpress?s[]=ajax
- there's only 1x reference to "ajax" in the phplist.org manual it referes to the Ethical Pet case study. Yeah, the one that migrated off from phplist..
- important, but a fucking stub. can we get an example listing 3x domains? https://resources.phplist.com/system/config/access_control_allow_origin?s[]=ajax
- the discuss thread above includes this example file from subscription_form_example.html (in subscription_form_example.html.zip) https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
- ugh, the first line includes the jquery src @ googleapis.com. Let's not make our users require decentraleyes to prevent google from tracking them on our site..
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
- it's looking like the best option is to use the no longer maintained "Newsletter Sign-Up" wp plugin still :\ I re-enabled it and added the line "[nsu_form]" to the lola paga again. I'll try to debug that before reinventing the wheel https://fef.opensourceecology.org/2015/09/03/lola/
- I checked the code; there's not even any JS; it's just a fucking form with submit action to our distinct phplist site (I figured it was some ajax cross-site rejection failure causing fall-back to the redirect). Maybe this plugin isn't what we want still..I disabled it again.
Sun Oct 14, 2018
- bought & installed my replacement ssd after my old one died unexpectedly
- install os, restore backups, etc
- catching up on old email
Mon Oct 01, 2018
- Note that my logs for the first few weeks of this quarter were lost when my laptop's ssd crashed. Please mind the gap.
