Revision as of 12:07, 3 December 2018

My work log from the year 2018 Quarter 4. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

Tue Nov 29, 2018

Marcin asked me to remove the address of OSE from our OSEmail template. No problem, but I reminded him that it's listed in many places all over the web https://wiki.opensourceecology.org/widki/OSEmail#2018-09

Had a call with Marcin, talked about discourse, phplist, and backblaze
1. Marcin recovered his passprhase for his ssh private key
2. Marcin was successfully able to mount the ose shared keepass file that lives on the hetzner2 ose server onto his local machine via sshfs
3. Marcin was able to decrypt the ose shared keepass file and gain access to our ose backblaze b2 credentials that I stored in the shared ose keepass file
4. Marcin was able to add the 2FA secret key for our backblaze b2 account into his phone's Google Authenticator app
5. Marcin was able to login to our backblaze b2 account using the user/pass in keepass + his google auth 2fa
6. Marcin was able to view the contens of our ose-server-backups bucket in the backblaze b2 wui
7. Marcin was able to add our billing information into our backblaze b2 account
8. Success!! Now I finally can migrate our backups from dreamhost onto backblaze b2

Wed Nov 28, 2018

sent an email to Marcin asking him to draft a new OSEmail for our upcoming repermission campaign
I read through this guide showing examples of repermission campaigns. my take aways: https://econsultancy.com/gdpr-examples-repermissioning-emails-campaigns/
1. We should make it as obvious as possible that the user must opt-in or (if they do nothing) they will be permanently blacklisted & receive no future newsletters
2. The subject should very clearly indicate that their action is required. I recommend: "Opt-In Action Required for OSEmail" or something similar
3. Rather than put the opt-in confirmation link as a hyperlink in the body of the text, we should actually make a huge green button titled "Yes, keep sending me the OSEmail newsletter" or something similar
4. The top of the email should be about the opt-in. The normal OSEmail update should go below the big green button

I spent some time crafting the above big green button html + fallback plaintext. Here's what I have so far

Hello friends,
<br/><br/>

Update from OSE: We have just setup phplist for our OSEmail newsletter. If you would like to keep recieving emails, you must <a href="http://phplist.opensourceecology.org/lists/?p=confirm&uid=43d035a41d2d893d21255158de6f9831">click this link</a>.

<a href="[CONFIRMATIONURL]" style="display:block; text-decoration:none; color: #FFFFFF;">
	<div style="text-align:center; margin: 50px">
	<span style="padding:15px; width: 100%; background-color: #03C03C; border-radius:15px; font-size: large; font-weight: bold;">
		Yes, keep sending me the OSEmail newsletter!
	</span>
	</div>
</a>
<br/><br/>

If you cannot see the link above, you can copy and paste the below link into your browser to keep recieving the OSEmail newsletter:
<br/><br/>
[CONFIRMATIONURL]

</div>

blah blah blah. some shit we did this past month. blah blah blah
<br/><br/>

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent nec mauris purus. Curabitur id lectus ornare, feugiat metus ut, porttitor nunc. Maecenas pharetra aliquet dictum. Vivamus vel vulputate ligula, non aliquam dolor. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Vestibulum finibus libero sed augue ornare, vel venenatis ligula porttitor. Integer sodales sed nisl eu suscipit. Nunc gravida vehicula sem, a congue massa venenatis eu. Morbi convallis urna a felis tincidunt, rutrum elementum sem varius. Vestibulum congue lectus nec ex venenatis sagittis. Quisque sed orci vitae mauris ultricies dignissim vel sit amet tortor. Aliquam imperdiet maximus enim, at posuere metus eleifend aliquet. Duis convallis ligula urna, eget hendrerit quam blandit at.
<br/><br/>

In eleifend porttitor quam eget sollicitudin. Vivamus egestas pellentesque nisi, sit amet porta nunc lobortis a. Proin interdum mi eu mauris aliquam ultricies. Sed lobortis sapien leo, porttitor vehicula purus molestie id. Morbi quis nunc id enim egestas efficitur pharetra sagittis tellus. Praesent imperdiet dolor urna, et porttitor eros dictum sit amet. Sed at fringilla turpis. Vivamus rhoncus porttitor massa, vitae venenatis lorem sagittis ac. Quisque quis erat sed magna elementum aliquet id non justo. Mauris cursus quam sed elit sodales, a pellentesque lorem euismod. Donec sodales sem non enim mattis volutpat. In auctor tincidunt posuere. Praesent posuere lectus quis elit accumsan, id congue ipsum euismod. Fusce molestie luctus metus, eu malesuada urna. Vivamus gravida dignissim porttitor. Nam sed diam efficitur, feugiat nulla id, tincidunt augue.
<br/><br/>

Nulla facilisi. Nunc vitae lobortis diam. Cras diam enim, finibus quis consequat finibus, rutrum et nisl. Morbi eleifend blandit dui, et eleifend mi viverra eget. Mauris ut congue massa. Aliquam erat volutpat. Suspendisse potenti. Praesent finibus ullamcorper ante, posuere consectetur neque bibendum eu. Sed ac sapien vitae augue blandit mattis. Pellentesque tincidunt sagittis mi at blandit. Sed tincidunt iaculis enim, volutpat accumsan felis imperdiet luctus. Cras metus leo, auctor ac purus sed, ornare mollis dui. Duis lobortis, libero id facilisis ullamcorper, tortor ipsum posuere purus, vel pellentesque ipsum urna eu nisl.
<br/><br/>

Quisque in ultricies mauris. Phasellus vel ante purus. Cras vel vestibulum ligula, vitae suscipit purus. Praesent at vehicula erat, varius blandit tellus. Vestibulum a mi vel lacus porttitor vestibulum in eu dolor. Duis eleifend purus iaculis massa tristique fermentum. Etiam vel velit nec nisi bibendum varius at viverra sapien.
<br/><br/>

In at tellus accumsan, egestas metus id, tempor nisi. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Vivamus eleifend elit dolor, vestibulum consectetur elit feugiat eget. Integer sed rutrum ante. Vivamus blandit neque sit amet mi pretium faucibus. Proin lectus ex, fringilla nec mi accumsan, maximus mattis turpis. In hac habitasse platea dictumst. Etiam sed tincidunt magna. Nullam purus turpis, tempor at elementum a, blandit nec massa.

...

I wasn't quite sure if the reason why I wasn't required to explicitly check the box agreeing to the Privacy Policy was because the user already existed. So, I deleted (not just blacklisted) myself from the phplist subscribers db entirely, then imported -> repermission campaign -> click confirmation url.
1. Still, I was unblacklisted without requiring to explicilty accepting the Privacy Policy. This is a huge issue!
I created a topic on the phplist discourse forms asking if there's some way to require the user to explicitly accept the ToS after the ToS https://discuss.phplist.org/t/re-permission-campaign-requiring-explicity-tos-action/4710

Tue Nov 27, 2018

I creado ated a new list in phplist called "OSEmail"
1. I checked the box to make it a "Public list"
2. this became list #3
it looks like attributes are configured seperately, so after creating the new list, I went to "Config" -> "Configure attributes" -> "Add new Attribute" to add a new field called "How did you find out about us?"
1. type = "textline"
2. I unchecked "Is this attriubte required?"
3. this became attribute #6
ok, so now I guess it's the "subscribe pages" that ties it all together. I created a new Subscribe Page with title = "Subscribe to our OSEmail Newsletter"
1. I left the default for "HTML Email choice" = "Offer checkbox for text"
2. I left the default for "Display email address confirmation field" = "Display email address confirmation field"
3. I left all the options in the "Transaction messages" tab at the defaults
4. In the "Select the attributes to use" tab, I checked the "Check this box to use this attribute in the page" for these attributes:
  1. attribute 4 = I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.
  2. attribute 6 = How did you find out about us?
5. In the "Select the lists to offer" tab, I checked only the box "OSEmail" list that I created above
this created the following subscribe page https://phplist.opensourceecology.org/lists/?p=subscribe&id=3
I validated that the OSEmail subscriber list is currently empty by the phplist wui @ "Subscribers" -> "Subscriber lists" -> "OSEmail" https://phplist.opensourceecology.org/lists/admin/?page=members&id=3
I copied the phplistAjax.php file to see if I could successfully alter it to work for this new subscriber list (page)
unfortunately, after changing the list's id from 2 to 3, it doesn't work...back to painful tracing

..

There's this guide to using xdebug for tracing https://devzone.zend.com/1135/tracing-php-applications-with-xdebug/
1. I tried & failed to do this back in August https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q3#Wed_Aug_22.2C_2018
I tried enabling xdebug.autotrace & setting xdebug.trace_output_dir in the vhost's apache config file, but no trace logs appeared
I tried adding the xdebug_start_trace() and xdebug_stop_trace() directly to the lists/index.php file, and still no trace logs were written
fucking xdebug, I give up again.

...

I changed the "Display email address confirmation field" radio button to "Don't display email address confirmation field", and the ajax form worked again.
so here's the updated ajax form page

<html><body>

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>

<script type="text/javascript">
function checkform() {

	// first, clear the response div from the previous attempts results
	jQuery("#result").empty();

	for (i=0;i<fieldstocheck.length;i++) {
		if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
			if (document.phplistSubscribeForm.elements[fieldstocheck[i]].checked) {
			} else {
				jQuery("#result").empty();
				alert("The following field is required:  "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
				return false;
			}
		} else {
			if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].value") == "") {
				alert("Please enter your "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");

				return false;
			}
		}
	}

	for (i=0;i<groupstocheck.length;i++) {
		if (!checkGroup(groupstocheck[i],groupnames[i])) {
			return false;
		}
	}
  
	if (! checkEmail()) {
		alert("Email address is not valid");
		return false;
	}

	return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name) {
	fieldstocheck[fieldstocheck.length] = value;
	fieldnames[fieldnames.length] = name;
}

var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name) {
	groupstocheck[groupstocheck.length] = value;
	groupnames[groupnames.length] = name;
}

function compareEmail() {
	return (document.phplistSubscribeForm.elements["email"].value == document.phplistSubscribeForm.elements["emailconfirm"].value);
}

function checkEmail() {
	var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.phplistSubscribeForm.elements["email"].value);
}

function checkGroup(name,value) {
	option = -1;
	for (i=0;i<document.phplistSubscribeForm.elements[name].length;i++) {
		if (document.phplistSubscribeForm.elements[name][i].checked) {
			option = i;
		}
	}

	if (option == -1) {
		alert ("Please enter your "+value);
		return false;
	}

	return true;
}

function submitForm() {

	// first, clear the response div from the previous attempts results
	jQuery("#result").empty();

	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#phplistSubscribeForm').serialize();
	jQuery.ajax( {

		type: 'POST',
		data: data,
		dataType: 'html',

		<!-- set the id in the url below (&id=X) to the Subscribe List ID, per
			 the "Config" -> "Subscribe pages" page in the phplist wui
			  * https://phplist.opensourceecology.org/lists/admin/?page=spage

			 Note that the Subscribe Page should use
			  * "Don't display email address confirmation field" or it will fail
			  * "Select the lists to offer" tab should check exactly one list

			 For more information, see
			  * https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
		-->
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=3',

		// defines a function that's called when our ajax call suceeds (ie: gets a
		// 200 response back)
		success: function (data, status, request) {

			jQuery("#result").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');

		 },

		// defines a function that's called when our ajax call fails (ie: gets a 500
		// response back)
		error: function (request, status, error) {

			jQuery("#result").empty();
			alert('Sorry, we were unable to process your subscription.');

		}

	});

}

</script>

<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">

	<p>Signup for our newsletter!</p>

	<!-- this simple form intentionally only requires the bare necessities:
	  [1] email address
	  [2] accept Privacy Policy
	-->
	<form method="post" action="" name="phplistSubscribeForm" id="phplistSubscribeForm">

		<!-- [1] email address -->
		<input type=text name=email required="required" placeholder="Email" size="30" id="email" />
		<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
		<br/>

		<!-- [2] accept Privacy Policy -->
		<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
		<label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a></label>
		<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE Privacy Policy");</script>
		<br/>

		<!-- other strange hidden inputs per phplist's ajax example thread
		  * https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
		-->
		<input type="hidden" name="list[2]" value="signup" />
		<input type="hidden" name="listname[2]" value="newsletter"/>
		<input type="hidden" name="htmlemail" value="1" />
		<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>

	</form>

	<!-- do some input sanity checking with js, then ajax() submit using jquery -->
	<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>

</div>

<!-- this input will fill-in with results from the phplist server after the ajax
submission via jquery -->
<div id="result"></div>


<script type="text/javascript">

// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';

</script>

</body></html>

...

I did a test import of users to the new list = OSEmail by the wui via "Subscribers" -> "Import subscribers" -> "Import by uploading a CSV file with email addresses and additional data.
1. ironicly, this procedure to import a "CSV" has a default delimiter of a TAB. I had to change this to an actual comma (,)
2. After clicking the "Import" button, I was prompted with a drop-down menu to match the columns to attributes. I used the column title "referrer" and selected the cooresponding attriubte name "How did you find out abou..."
3. Here's the data I used:

user@personal:~/tmp/ose/phplist$ cat subscribeListTest.20181127.csv 
email,referrer
michael@opensourceecology.org,my friend travis
phplist@opensourceecology.org,marcin's ted talk from 2011
user@personal:~/tmp/ose/phplist$

1. after I confirmed the import, I got the following email from phplist@opensourceecology.org

0 emails succesfully imported to the database and added to 1 lists.
2 emails subscribed to the lists
2 emails already existed in the database
Subscriber data was updated for 2 subscribers
0 subscribers were matched by foreign key, 2 by email

I noted that the imported subscribers did *not* recieve an email stating that they were added. This is what we want at this time, but not generally what we want when, for example, we manually import a list of subscribers from a sign-up sheet (data entry from a workshop or talk)

...

I went ahead and clicked the "Enable" button for the plugin = "inviteplugin" via the phplist wui at "Config" -> "Mange plugins"
I confirmed that both the subscribers imported above were not blacklisted and were subscribed to the OSEmail list
I sent a new campaign, where I set the "Send as" = "Invite" (as opposed to "HTML" or "Text") on the Formait tab
After I processed the repermission campagin, I got the email for both email accounts.
I re-checked the subscribers, and I found that they were both blacklisted now (the OSEmail list was still listed as subscribed, but the blacklist overrides this)
Without clicking the confirmation link in the repermission campaign's email, I tried to send another campaign to OSEmail. I confirmed that neither of the 2x subscribers (still blacklisted) recieved the follow-up campaign.
I went ahead and clicked the confirmation link in one of the email providers http://phplist.opensourceecology.org/lists/?p=confirm&uid=43d035a41d2d893d21255158de6f9831
1. When I loaded the above URL, I got a "Thank you for confirming your subscription to our newsletter..." message
2. Also, after clicking the link, I got a "Welcome to our Newsletter" email
I checked the subscriber again, and now the user wasn't listed as blacklisted
digging deeper, I clicked the "History" button on the subscriber's profile, and the "Subscription" tab had the following entries
1. 27 November 2018 20:33:53 Import by maltfield: http://phplist.opensourceecology.org/lists/admin/?page=user&id=2 No user details changed
2. 27 November 2018 20:44:58 Added to blacklist: Added to blacklist for reason Blacklisted by the invitation plugin
3. 27 November 2018 21:11:15 Removed from blacklist: Removed from blacklist
4. 27 November 2018 21:11:15 Confirmation: Subscriber removed from do-not-send list for manual confirmation of subscription
5. 27 November 2018 21:11:15 Confirmation: Lists: *newsletter *OSEmail
therefore, we're making note (even for existing users) of when:
1. we imported the subscriber into phplist (and which user did it)
2. we blacklisted them via the invite plug as part of the repermission campaign to ensure gdpr compliance
3. when the user clicked the link requesting to confirm their subscription. It also explicilty lists the URL of their confirmation link, ie: REQUEST_URI = /lists/?p=confirm&uid=43d035a41d2d893d21255158de6f9831
note that I wasn't forced to check a box agreeing to the new Priacy Policy--which is the whole point of the re-permission campaign! This must be fixed!

Mon Nov 26, 2018

discovered that our letsencrypt certifcate is expired on the prod site! It is now " November 26, 2018, 4:41" and it expired on " November 26, 2018, 4:41".
we should probably have an automated alert go out when the cert is about to expire in less than 7 days.
the log at /var/log/letsEncryptRenew.log shows errors from the phplist domain. This is probably a result of the port change from 4443 to 443.
I updated the letsencrypt config file for the opensourceecology.org domain

[root@hetzner2 cron.d]# grep phplist /etc/letsencrypt/renewal/opensourceecology.org.conf 
phplist.opensourceecology.org = /var/www/html/certbot/htdocs
[root@hetzner2 cron.d]# vim /etc/letsencrypt/renewal/opensourceecology.org.conf 
[root@hetzner2 cron.d]# grep phplist /etc/letsencrypt/renewal/opensourceecology.org.conf 
phplist.opensourceecology.org = /var/www/html/phplist.opensourceecology.org/public_html
[root@hetzner2 cron.d]#

I re-ran the cron job, and that fixed the websites

[root@hetzner2 cron.d]# cat /etc/cron.d/letsencrypt 
# once a month, update our letsencrypt cert
20 4 13 * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log
[root@hetzner2 cron.d]# /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log
[root@hetzner2 cron.d]#

Sat Nov 24, 2018

I fixed the phplist email template overflow issue by appending 'width: 600px; overflow: hidden;" to the div in the td surrounding the [CONTENT] https://wiki.opensourceecology.org/wiki/File:OSEmail_phplist.tar.gz

...

I sent Marcin an email about adding the billing info to our Backblaze B2 account on 10/20. It's been over a month, so I sent Marcin a follow-up again--asking if we should schedule a call so I can assist him in this task.
I logged into our backblaze b2 account to check the status of the file uploads (currently just /etc/ as we're space limited with our free account)
1. it looks like the data retention policy is only partially working properly: the monthly files are being kept as desired, but we currently have 8 daily files. This confirms my confusion from 4 months ago, when I first setup the lifecycle settings https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q3#Sat_Jul_28.2C_2018

I created the above lifecycle settings in the wui. It was easy, after logging in, just click on "buckets" on the left navigation panel, then click the "Lifecycle Settings" link on the corresponding bucket (ie: ose-server-backups). Then there's an easily dialog that opens. Click the "Use custom lifecycle rules" option, and "Add Lifecycle Rules", which will add another row to the list of rules. Each rule has 3 fields: "File Path (fileNamePrefix)", "Days Till Hide (daysFromUploadingToHiding)", and "Days Till Delete (daysFromHidingToDeleting)". I set both fields to the numbers as described above. I'm not sure if this means that daily files will be deleted in 3 or 6 days (the sum of the two?). I'll just check back later and see. Note that it wouldn't let me set either of these fields to "0".

1. interstingly, 4/8 of the files that show in the wui don't show from the cli. also, there's no weekly files at all. hmm. here's a list of all the files currently on the server

[b2user@hetzner2 B2_Command_Line_Tool]$ ~/virtualenv/bin/b2 ls ose-server-backups
daily_hetzner2_20181121_091749.tar.gpg
daily_hetzner2_20181122_091803.tar.gpg
daily_hetzner2_20181123_091819.tar.gpg
daily_hetzner2_20181124_091828.tar.gpg
monthly_hetzner2_20181001_091809.tar.gpg
monthly_hetzner2_20181101_091810.tar.gpg
[b2user@hetzner2 B2_Command_Line_Tool]$

1. I added a stub Backblaze B2 section next to our glacier section in the OSE_Server wiki page https://wiki.opensourceecology.org/wiki/OSE_Server#Restore_from_Backblaze_B2
2. I confirmed that I can download the "hidden" files from the wui, which were missing from the cli. I think this "Days Till Hide" = daysFromUploadingToHiding *should* be set to 0
  1. as mentioned before, this can't actually be set to 0. Also, it looks like these lifecycle settings cannot be updated. So I went ahead and deleted the existing "daily_" lifecycle rule with "daysFromUpladingToHiding=3 + daysFromHidingToDeleting=3" to "daysFromUpladingToHiding=2 + daysFromHidingToDeleting=1" via the wui. That should give us the data retention of 3 daily backups at a time, but I'll verify this in the future.
3. I also went ahead and updated the lifecycle settings for "weekly_" and "monthly_" files as well so that the "Days Till Delete" = daysFromHidingToDeleting = 1.
4. to summarize, the new rules are:

fileNamePrefix|daysFromUploadingToHiding|daysFromHidingToDeleting
daily_|2|1|
weekly_|30|1
monthly_364|1

1. it looks like there's a logic error in the script preventing "weekly_" files from being created; it matches on "01", but it should match on "1"

# determine the prefix/interval of today's backup archive                                                                    
if  "`date +"%m%d"`" == "0101" ; then                                                                                    
   # today is January 1st                                                                                                    
   archivePrefix="yearly"                                                                                                    
elif  `date +"%d"` == '01' ; then                                                                                        
   # today is the 1st of the month                                                                                           
   archivePrefix="monthly"                                                                                                   
elif  `date +"%u"` == '01' ; then                                                                                        
   # today is the first day of the week = Monday                                                                             
   archivePrefix="weekly"                                                                                                    
else                                                                                                                         
   # otherwise, we'll just call this a "daily" backup                                                                        
   archivePrefix="daily"                                                                                                     
fi

1. but note that the `date +"%u"` command does not have a leading 0

[maltfield@hetzner2 ~]$ date +"%u"
6
[maltfield@hetzner2 ~]$

1. I changed the match to " == '1'". I'll verify that this is working in a month or so

..

began investigating the repermission campaign solution. it looks like we need the "invite" pluigin https://discuss.phplist.org/t/gdpr-re-permission-campaigns-invite-plugin/4086
I confirmed that we already have the invite plugin installed (looks like it came by default), but it's not enabled https://phplist.opensourceecology.org/lists/admin/?page=plugins
here's the invite plugin page on phplist.com https://resources.phplist.com/plugin/invite
and here's the source code (with README) for the invite plugin on github https://github.com/phpList/phplist-plugin-invite
based on what I read above, I think this will be our procedure:
1. create a new list
2. import our subscribers from google spreadsheets into phplist, checking the box to *not* send them an email asking for them to confirm the subscription
3. send out a new campaign (ideally as an actual OSEmail with content per Marcin's request) to all the subscribers, making sure to check the "invite" radio button on the Format tab. At the top of the campaign's [CONTENT], we should state that this is our first OSEmail from our phplist install, and that users will not recieve future updates unless they click the link (href=[CONFIRMATIONURL]).
  1. I have to confirm that, at some point in this process, the user is required to tick a box that they agree with our Privacy Policy (linking to the Privacy Policy on our wiki)
4. the phplist manual specifically calls out repermission campaigns for gpdr https://www.phplist.org/manual/ch048_gdpr.xhtml
  1. it links to this article https://econsultancy.com/gdpr-examples-repermissioning-emails-campaigns/

Fri Nov 23, 2018

I updated our phplist email template with actual social media links for twitter & facebook. rather than do a 'share this newsletter' approach, I just made it a generic "follow" or "like"
1. twitter link is always now https://twitter.com/intent/user?screen_name=osecology
2. facebook apparently doesn't have an equivalent without including scripts from their site. fuck that. the link is always now just pointing to the facebook page with no action https://www.facebook.com/OpenSourceEcology/
I also removed the "web version" link
I updated the template as stored on the wiki https://wiki.opensourceecology.org/wiki/File:OSEmail_phplist.tar.gz

worked on a change to the phplist codebase per the "hack attempt" issues discussed in this thread https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-subscribelib2-php/4578/11
1. submitted a PR https://github.com/phpList/phplist3/pull/444
updated our privacy policy to include newsletters https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy#Newsletters

I made some further improvements to the ajax form

<html><body>

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>

<script type="text/javascript">
function checkform() {

	// first, clear the response div from the previous attempts results
	jQuery("#result").empty();

	for (i=0;i<fieldstocheck.length;i++) {
		if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
			if (document.phplistSubscribeForm.elements[fieldstocheck[i]].checked) {
			} else {
				jQuery("#result").empty();
				alert("The following field is required:  "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
				return false;
			}
		} else {
			if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].value") == "") {
				alert("Please enter your "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");

				return false;
			}
		}
	}

	for (i=0;i<groupstocheck.length;i++) {
		if (!checkGroup(groupstocheck[i],groupnames[i])) {
			return false;
		}
	}
  
	if (! checkEmail()) {
		alert("Email address is not valid");
		return false;
	}

	return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name) {
	fieldstocheck[fieldstocheck.length] = value;
	fieldnames[fieldnames.length] = name;
}

var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name) {
	groupstocheck[groupstocheck.length] = value;
	groupnames[groupnames.length] = name;
}

function compareEmail() {
	return (document.phplistSubscribeForm.elements["email"].value == document.phplistSubscribeForm.elements["emailconfirm"].value);
}

function checkEmail() {
	var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.phplistSubscribeForm.elements["email"].value);
}

function checkGroup(name,value) {
	option = -1;
	for (i=0;i<document.phplistSubscribeForm.elements[name].length;i++) {
		if (document.phplistSubscribeForm.elements[name][i].checked) {
			option = i;
		}
	}

	if (option == -1) {
		alert ("Please enter your "+value);
		return false;
	}

	return true;
}

function submitForm() {

	// first, clear the response div from the previous attempts results
	jQuery("#result").empty();

	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#phplistSubscribeForm').serialize();
	jQuery.ajax( {

		type: 'POST',
		data: data,
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
		dataType: 'html',

		// defines a function that's called when our ajax call suceeds (ie: gets a
		// 200 response back)
		success: function (data, status, request) {

			jQuery("#result").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');

		 },

		// defines a function that's called when our ajax call fails (ie: gets a 500
		// response back)
		error: function (request, status, error) {

			jQuery("#result").empty();
			alert('Sorry, we were unable to process your subscription.');

		}

	});

}

</script>

<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">

	<p>Signup for our newsletter!</p>

	<!-- this simple form intentionally only requires the bare necessities:
	  [1] email address
	  [2] accept Privacy Policy
	-->
	<form method="post" action="" name="phplistSubscribeForm" id="phplistSubscribeForm">

		<!-- [1] email address -->
		<input type=text name=email required="required" placeholder="Email" size="30" id="email" />
		<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
		<br/>

		<!-- [2] accept Privacy Policy -->
		<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
		<label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a></label>
		<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE Privacy Policy");</script>
		<br/>

		<!-- other strange hidden inputs per phplist's ajax example thread
		  * https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
		-->
		<input type="hidden" name="list[2]" value="signup" />
		<input type="hidden" name="listname[2]" value="newsletter"/>
		<input type="hidden" name="htmlemail" value="1" />
		<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>

	</form>

	<!-- do some input sanity checking with js, then ajax() submit using jquery -->
	<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>

</div>

<!-- this input will fill-in with results from the phplist server after the ajax
submission via jquery -->
<div id="result"></div>


<script type="text/javascript">

// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';

</script>

</body></html>

fixed an issue where sending a new campaign redirected to the site on port 4443 (broken) by commenting-out the line defining the HTTP_HOST with the 4443 port
fixed a modsecurity false-positive
1. 981317 sqli
sent a test campaign
1. found an issue that really long words result in an overflow not word-wrapping, causing the header ose logo image from moving to the right.

Thr Nov 22, 2018

Duncan responded to my thread about the "hack attempt" issue. He found a very obsecure config issue: there was no lists checked in the "lists to offer" section of the Subscribe Page. I suggested a code change to emit an error before the "exit" to make this issue easier to pinpoint https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-subscribelib2-php/4578/9
I made some changes to the phplist ajax form, adding some css styles and html comments. I should still probably set the encoding and run it through a w3c checker to ensure compliance, but here's the current state of it

<html><body>

<style type="text/css">

	#phplistSubscribeForm {
		background-color: #f3f3f3;
		border: solid 1px #a1a1a1;
		padding: 10px;
		//width: 600px;
	}
    
	#phplistSubscribeForm label, #phplistSubscribeForm input {
		display: block;
		width: 300px;
		float: left;
		margin-bottom: 10px;
	}
 
	#phplistSubscribeForm label {
		text-align: right;
		padding-right: 20px;
	}
 
	br {
		clear: left;
	}

</style>

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>

<script type="text/javascript">
function checkform() {

	// first, clear the response div from the previous attempts results
	jQuery("#result").empty();

	for (i=0;i<fieldstocheck.length;i++) {
		if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
			if (document.phplistSubscribeForm.elements[fieldstocheck[i]].checked) {
			} else {
				jQuery("#result").empty();
				alert("The following field is required:  "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
				return false;
			}
		} else {
			if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].value") == "") {
				alert("Please enter your "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");

				return false;
			}
		}
	}

	for (i=0;i<groupstocheck.length;i++) {
		if (!checkGroup(groupstocheck[i],groupnames[i])) {
			return false;
		}
	}
  
	if (! checkEmail()) {
		alert("Email address is not valid");
		return false;
	}

	return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name) {
	fieldstocheck[fieldstocheck.length] = value;
	fieldnames[fieldnames.length] = name;
}

var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name) {
	groupstocheck[groupstocheck.length] = value;
	groupnames[groupnames.length] = name;
}

function compareEmail() {
	return (document.phplistSubscribeForm.elements["email"].value == document.phplistSubscribeForm.elements["emailconfirm"].value);
}

function checkEmail() {
	var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.phplistSubscribeForm.elements["email"].value);
}

function checkGroup(name,value) {
	option = -1;
	for (i=0;i<document.phplistSubscribeForm.elements[name].length;i++) {
		if (document.phplistSubscribeForm.elements[name][i].checked) {
			option = i;
		}
	}

	if (option == -1) {
		alert ("Please enter your "+value);
		return false;
	}

	return true;
}

function submitForm() {

	// first, clear the response div from the previous attempts results
	jQuery("#result").empty();

	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#phplistSubscribeForm').serialize();
	jQuery.ajax( {

		type: 'POST',
		data: data,
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
		dataType: 'html',

		// defines a function that's called when our ajax call suceeds (ie: gets a
		// 200 response back)
		success: function (data, status, request) {

			jQuery("#result").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');

		 },

		// defines a function that's called when our ajax call fails (ie: gets a 500
		// response back)
		error: function (request, status, error) {

			jQuery("#result").empty();
			alert('Sorry, we were unable to process your subscription.');

		}

	});

}

</script>

<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">

	<!-- this simple form intentionally only requires the bare necessities:
	  [1] email address
	  [2] accept Privacy Policy
	-->
	<form method="post" action="" name="phplistSubscribeForm" id="phplistSubscribeForm">

		<!-- [1] email address -->
		<label for="email">Email address *</label>
		<input type=text name=email required="required" placeholder="" size="40" id="email" />
		<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
		<br/>

		<!-- [2] accept Privacy Policy -->
		<label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label>
		<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
		<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script>
		<br/>

		<!-- other strange hidden inputs per phplist's ajax example thread
		  * https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
		-->
		<input type="hidden" name="list[2]" value="signup" />
		<input type="hidden" name="listname[2]" value="newsletter"/>
		<input type="hidden" name="htmlemail" value="1" />
		<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>
		<br/>

	</form>

	<!-- do some input sanity checking with js, then ajax() submit using jquery -->
	<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>

</div>

<!-- this input will fill-in with results from the phplist server after the ajax
submission via jquery -->
<div id="result"></div>


<script type="text/javascript">

// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';

</script>

</body></html>

Sun Nov 07, 2018

Duncan Cameron noted that in their environment, my change to throw an exception didn't change the exit code to 200. I looked it up, and found that if display_errors is enabled, then it will stay at 200--otherwise it's 500. https://github.com/phpList/phplist3/pull/432
I replied to this thread https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-subscribelib2-php/4578

Sun Nov 04, 2018

the phplist team approved & merged my PR to throw an exception when the ajax subscribe fails on the server, which changes the https status code from 200 ok to 500 internal server error https://github.com/phpList/phplist3/pull/432
continuing with why the subscribelib2.php script thinks our ajax subscribe is a "hack attempt" and exits prematurely..
it looks like it's because the the key in the key/value pair from the "list[2]=subscribe" argument is empty.

error_log( "made it to AC" );                                                                                                           
                                                                                                                                        
	if (isset($_POST['list']) && is_array($_POST['list'])) {                                                                            
		foreach ($_POST['list'] as $key => $val) {                                                                                      
error_log( "\tlist key|$key| val|$val|" );                                                                                                          if ($val == 'signup') {                                                                                                     
				$key = sprintf('%d', $key);                                                                                             
				if (!empty($key)) {                                                                                                     	
error_log( "made it to AAA" );                                                                                                          
					$result = Sql_query(sprintf('replace into %s (userid,listid,entered) values(%d,%d,now())',                          
						$GLOBALS['tables']['listuser'], $userid, $key));                                                                
					$lists .= "\n  * ".listname($key);                                                                                  
					$subscriptions[] = $key;                                                                                            
                                                                                                                                        
					addSubscriberStatistics('subscribe', 1, $key);                                                                      
				} else {                                                                                                                
					//# hack attempt...                                                                                                 
error_log( "made it to AAB" );                                                                                                          
					exit;                                                                                                               
				}                                                                                                                       
			}                                                                                                                           
		}                                                                                                                               
	}                                                                                                                                   
error_log( "made it to AD" );

the above code yields the following in the error log, indicating that the loop itreates through two distinct elements in the $_POST['list'] array. One's key is '2' and the other is empty.

[Sun Nov 04 18:56:57.061936 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AC, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
[Sun Nov 04 18:56:57.061951 2018] [:error] [pid 14919] [client 127.0.0.1:44604] \tlist key|2| val|signup|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
[Sun Nov 04 18:56:57.061977 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AAA, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
[Sun Nov 04 18:56:57.069666 2018] [:error] [pid 14919] [client 127.0.0.1:44604] \tlist key|| val|signup|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
[Sun Nov 04 18:56:57.069708 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AAB, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php

I created a thread on the phplist discourse forms asking why that "hack attempt" exit exists, and describing how it's breaking our ajax subscription flow via jquery.ajax(). I asked if I can remove it in a PR if it's not necessary https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-libsubscribe2-php/4578
upon further research, I found that my ajax form is indeed submitting only "list[2]=signup" and that the "list[]=signup" paramater is actually being added by phplist (!) here https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/index.php#L247-L249
1. I'm not sure what the $GLOBALS['pagedata']['lists'] data *should* have. In any case, I updated the discourse thread linked above
in the meantime, I commented-out the "hack attempt" comment and encapsulating else statement
now the subscribelib2.php code is ending at the "return 2" line here https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/admin/subscribelib2.php#L407
ok, so now when I send the right inputs, the result div gets populated with the "thanks, you have been added to the newsletter..." message as returned by the phplist server's response. Horray! I also confirmed that when I submit with bad data, it fails properly (with a pop-up saying "Sorry, we were unable to process your subscription."
it doesn't actually clear the success message after the fail message comes back, so that needs to be fixed next week

...

I did some further digging into the code that checks for required attributes, regarding this bug I submitted earlier this week https://mantis.phplist.org/view.php?id=19503
it appears that the first part of the if statement checks $subscribeagedata rather than checking the DB. I'm not sure what the advantage of this is, except for maybe making some attributes optional on some subscribe pages. The issue in my case was that my attribute in the relevant subscribe page (as determined by the GET variable (id=2) had the ToS attribute's "Is this attribute required" checkbox ticked, but then the "Check this box to use this attribute in the page" checkbox was *not* ticked. https://phplist.opensourceecology.org:4443/lists/admin/?page=spageedit&tk=2d2c23511c2b9762c09026d77c25cd31&id=2
when I _did_ check the "use this attribute" checkbox (in addition to the "required" checkbox, which was already checked), the $subscribepagedata finally included the ToS attribute4 in the list of required attributes

[Sun Nov 04 19:41:22.044846 2018] [:error] [pid 30671] [client 127.0.0.1:58448] subscribepagedata:|Array\n(\n    [ajax_subscribeconfirmation] => <h3>Thanks, you have been added to our newsletter</h3><p>You will receive an email to confirm your subscription. Please click the link in the email to confirm</p>\n    [attribute004] => 4######0###1\n    [attributes] => 4+\n    [button] => Subscribe to the selected newsletters\n    [emaildoubleentry] => no\n    [footer] => </div>\r\n</div>\r\n</div><!-- ENDOF #mainContent-->\r\n</div><!-- ENDOF .wrapper -->\r\n</div><!-- ENDOF #container -->\r\n\r\n<div id="footer">\r\n<div id="footerframe">\r\n<ul class="list-unstyled">\r\n<li> </li>\r\n</ul>\r\n</div>\r\n</div>\r\n<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>\r\n<script type="text/javascript" src="admin/js/phplistapp.js"></script>\r\n<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>\r\n<script>\r\n/* do not remove this from here */\r\n$(document).ready(function(){\r\n    if ( $('body').hasClass('invisible') ){\r\n        myfunction();\r\n    }\r\n\t$('#edit_list_categories input.form-control').tagsinput('refresh');\r\n\r\n});\r\n</script>\r\n</body>\r\n\r\n</html>\r\n\n    [header] => <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />\r\n<meta name="theme-color" content="#2C2C2C"/>\r\n<link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" />\r\n<link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" />\r\n<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />\r\n</head>\r\n\r\n<body class="fixed invisible">\r\n\r\n<div id="container">\r\n\r\n<div id="header" class="navbar navbar-inverse">\r\n<div id="rack-functions" class="container">\r\n\t<h1 id="logo"><a href="http://phplist.opensourceecology.org" title="Visit our website">Open Source Ecology</a></h1>\r\n</div>\r\n</div>\r\n<div id="wrapper" class="container">\r\n<div id="mainContent">\r\n\r\n<div class="panel">\r\n<br />\r\n<div class="content  well">\n    [htmlchoice] => htmlonly\n    [intro] => <h3>Subscribe to our newsletters</h3>\n    [language_file] => \n    [thankyoupage] => <h3>Thank you for subscribing to our newsletters.</h3>\r\nYour email address has been added to our system. You will receive a message with a request to confirm your membership. Please make sure to click the link in that message to confirm your subscription.\n    [title] => poc\n    [lists] => \n    [rssdefault] => \n    [rssintro] => \n    [rss] => \n)\n|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php

I updated the bug report with my UX change suggestion

Fri Nov 02, 2018

when I was poking at the ajax form & server-side code, I found that the ajax's "error" function was successfully called when I had a php-side syntax error, causing a 500 (internal server error) to be returned. So that's proven to work. It still does not, however, catch the FAIL response, which is included in a 200 response. I supose I could either fix the php code to return a non-200 (and submit a PR) or just catch FAIL messages from the ajax success() function and then call the ajax error() function from within success().
1. I decided that the best way to handle this is for the php script to throw an exception instead of just print "FAIL", which causes a 500 status to be returned & is caught by the AJAX code
2. I created a bug report for this here https://mantis.phplist.org/view.php?id=19524
ugh, I discovered that phplist created their own sqllog() function in mysqli.inc, and then hard-codes log files to be written to "/tmp/". I wrote a post to the phplist Discourse forum askint the developers why they avoid php's built-in error_log() function https://discuss.phplist.org/t/why-doesnt-phplist-use-error-log/4570
1. Xheni responded stating that they'd rather not spam the error log, and instead create a new log file in a new dir and lock it down with php. I stated that I don't think that's really possible, and that using trigger_error() would allow you to set messages as E_STRICT--which actually would not be written to the log file by default (only done so if the logging verbosity was increased on, say, a dev server). I also mentioned that they should use the php function sys_get_temp_dir() to get php's upload_tmp_dir rather than expect to be able to access '/tmp' as it's hardcoded now--since open_basedir may deny access to '/tmp/'.
2. I reiterated that using trigger_error() would be able to send the messages to a log file that was readable only to root, something that I don't think would be possible to achieve within a php script.
I finally found out what the form input VerificationCodeX is for. Apparently we just create it as a hidden field and do nothing with it. I gues the hope is that spam bot will attempt to fill it in. If it's anything but empty, we reject it as spam. I do something similar with an input field on email.michaelaltfield.net https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/admin/subscribelib2.php#L92-L119
for some reason when subscfribelib2.php encounters an error, it returns lots of info, including the sucess messages. But when it successfully adds my user, it returns a generic payload without the success message.

<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phpList version 3.3.3" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" href="./images/phplist.ico" />

it appears that subscribelib2.php ends early (exit? return?). I should find out why. The goal here is to get phplist to return a 200 with the phplist configured success message on success. And it should return a 500 on failure, hopefully with some message as to why (though I'd accept a missing explination at this point)
looks like I'm being detected as a hacker? I found that It's exiting just after the line "hack attempt" further research is required. https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/admin/subscribelib2.php#L255-L256

	if (isset($_POST['list']) && is_array($_POST['list'])) {                                                                                
		foreach ($_POST['list'] as $key => $val) {                                                                                          
			if ($val == 'signup') {                                                                                                         
				$key = sprintf('%d', $key);                                                                                                 
				if (!empty($key)) {                                                                                                         
error_log( "made it to AAA" );                                                                                                              
					$result = Sql_query(sprintf('replace into %s (userid,listid,entered) values(%d,%d,now())',                              
						$GLOBALS['tables']['listuser'], $userid, $key));                                                                    
					$lists .= "\n  * ".listname($key);                                                                                      
					$subscriptions[] = $key;                                                                                                
                                                                                                                                            
					addSubscriberStatistics('subscribe', 1, $key);                                                                          
				} else {                                                                                                                    
					//# hack attempt...                                                                                                     
error_log( "made it to AAB" );                                                                                                              
					exit;                                                                                                                   
				}                                                                                                                           
			}                                                                                                                               
		}                                                                                                                                   
	}                                                                                                                                       error_log( "made it to AD" );

Tue Oct 30, 2018

continuing with my quest to get the ajax form to work properly..
I thought maybe the root cause of the empty error was the js console's message about CORS issues

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at ‘https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2’. (Reason: Credential is not supported if the CORS header ‘Access-Control-Allow-Origin’ is ‘*’).
</b>
# I changed the phplist config file from allowing '*' to just our domain
<pre>
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );

but then the error changed to one about the Credentials issue

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2. (Reason: expected ‘true’ in CORS header ‘Access-Control-Allow-Credentials’).

I did some research on this specific error, and it appears to pop-up when both the server and the client don't explicitly permit the Credentials flag, which would allow the server to set cookies across domains, creating possible CSRF vulnerabilities https://stackoverflow.com/questions/24687313/what-exactly-does-the-access-control-allow-credentials-header-do
I checked the debugger for cookies being stored, and--sure enough--there was one named 'browsetrail'. I searched the entire phplist vhost dir for this file, and found a few releavant files. Namely, I found one of the js files we were including at the top of our script = phplistapp.js

[root@hetzner2 phplist.opensourceecology.org]# grep -irl 'browsetrail' *
public_html/lists/admin/connect.php
public_html/lists/admin/ui/dressprow/js/all.js
public_html/lists/admin/ui/dressprow/js/all.min.js
public_html/lists/admin/ui/default/js/all.js
public_html/lists/admin/ui/default/js/all.min.js
public_html/lists/admin/js/phplistapp.js
[root@hetzner2 phplist.opensourceecology.org]#

so I commented-out the include of this in our script, and the CORS erros went away entirely

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
                                                                                                                                            
<!-- trying to prevent the 'browsetrail' cookie causing CORS errors in Access-Control-Allow-Credentials
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
-->
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

After some digging, I found strikingly little logic behind the ajax backend that's listening to our form's requests. It's just this https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/index.php#L242-L272

			case 'asubscribe': //# subscribe with Ajax
				$_POST['subscribe'] = 1;
				if (isset($_GET['email']) && !isset($_POST['email'])) {
					$_POST['email'] = $_GET['email'];
				}
				foreach (explode(',', $GLOBALS['pagedata']['lists']) as $listid) {
					$_POST['list'][$listid] = 'signup';
				}
				$_POST['htmlemail'] = 1; //# @@ should actually be taken from the subscribe page data
				$success = require 'admin/subscribelib2.php';
				$result = ob_get_contents();
				ob_end_clean();
				if (stripos($result, $GLOBALS['strEmailConfirmation']) !== false ||
					stripos($result, $pagedata['thankyoupage']) !== false
				) {
					if (!empty($pagedata['ajax_subscribeconfirmation'])) {
						$confirmation = $pagedata['ajax_subscribeconfirmation'];
					} else {
						$confirmation = getConfig('ajax_subscribeconfirmation');
					}
					if (empty($confirmation)) {
						echo 'OK';
					} else {
						echo $confirmation;
					}
					exit;
				} else {
					echo 'FAIL';
				}
				break;

so basically all I have to work with is if it responded with 'FAIL' or 'OK'. In both cases, the jquery ajax function will hook the "success" function.
1. this is very bad as we're basically depending on our js form for proving that the user accepted the Privacy Policy. Indeed, if I remove the JS on the form that ensures the "I agree..." checkbox is ticked and submit the form with it unticked, then phplist happily subscribes the user! This is *wrong*!
the best solution, then, is to update this code
1. unfortunaetly, the ajax solution that we're updating is being deprecated in the next release of phplist4 in favor of the REST API
2. the attribute that I need to verify identifies itself as "attribute4" — which is hardly specific
3. note, however, that the attribute does have the property (meta attribute?) "is this attribute required"
4. therefore, probably the best way to write this robustly is to get a list of all the attributes that are requried & ensure that they are present. If any required attributes are not present (or have 0 /blank/ null values), then do *not* subscribe the user and return an error
so, actually, the block above does require another script--which actually does the logic for adding the user. Indeed, there is some logic in there for checking for required fields. https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/admin/subscribelib2.php#L18=L81

//# Check if input is complete
$allthere = 1;
$subscribepagedata = PageData($id);
if (isset($subscribepagedata['language_file']) && is_file(dirname(FILE).'/../texts/'.basename($subscribepagedata['language_file']))) {
	@include_once dirname(FILE).'/../texts/'.basename($subscribepagedata['language_file']);
}
// Allow customisation per installation
if (is_file($_SERVER['DOCUMENT_ROOT'].'/'.basename($GLOBALS['language_module']))) {
	include_once $_SERVER['DOCUMENT_ROOT'].'/'.basename($GLOBALS['language_module']);
}
if (!empty($data['language_file']) && is_file($_SERVER['DOCUMENT_ROOT'].'/'.basename($data['language_file']))) {
	include_once $_SERVER['DOCUMENT_ROOT'].'/'.basename($data['language_file']);
}
$required = array();   // id's of missing attribbutes '
if (count($subscribepagedata)) {
	$attributes = explode('+', $subscribepagedata['attributes']);
	foreach ($attributes as $attribute) {
		if (isset($subscribepagedata[sprintf('attribute%03d',
					$attribute)]) && $subscribepagedata[sprintf('attribute%03d', $attribute)]
		) {
			list($dummy, $dummy2, $dummy3, $req) = explode('###',
				$subscribepagedata[sprintf('attribute%03d', $attribute)]);
			if ($req) {
				array_push($required, $attribute);
			}
		}
	}
} else {
	$req = Sql_Query(sprintf('select * from %s', $GLOBALS['tables']['attribute']));
	while ($row = Sql_Fetch_Array($req)) {
		if ($row['required']) {
			array_push($required, $row['id']);
		}
	}
}
if (count($required)) {
	$required_ids = implode(',', $required);
	// check if all required attributes have been entered;
	if ($required_ids) {
		$res = Sql_Query("select * from {$GLOBALS['tables']['attribute']} where id in ($required_ids)");
		$allthere = 1;
		$missing = '';
		while ($row = Sql_Fetch_Array($res)) {
			$fieldname = 'attribute'.$row['id'];
			$thisonemissing = 0;
			if ($row['type'] != 'hidden') {
				$thisonemissing = empty($_POST[$fieldname]);
				if ($thisonemissing) {
					$missing .= $row['name'].', ';
				}
				$allthere = $allthere && !$thisonemissing;
			}
		}
		$missing = substr($missing, 0, -2);
		if ($allthere) {
			$missing = '';
		}
	}
} else {
	$missing = '';
}

so the "$requried" var should store a list of all the arguments that are required. I added some test debug code to output this array, and it only contained one element = "1", which I'm assuming is the email address. further research is required
also, I realized that there doesn't appear to be a per-list attribute config section, which makes me think that you can't have distinct attribute sets for different subscriber lists, which may be a problem. Further research into this is required

...

I dug into the db further to see what I could dig up: I found only 1 row in the table 'phplist_user_attribute' with my 'termsofservice' attribute, which is indeed marked as required

MariaDB [phplist_db]> describe phplist_user_attribute;
+---------------+--------------+------+-----+---------+----------------+
| Field         | Type         | Null | Key | Default | Extra          |
+---------------+--------------+------+-----+---------+----------------+
| id            | int(11)      | NO   | PRI | NULL    | auto_increment |
| name          | varchar(255) | NO   | MUL | NULL    |                |
| type          | varchar(30)  | YES  |     | NULL    |                |
| listorder     | int(11)      | YES  |     | NULL    |                |
| default_value | varchar(255) | YES  |     | NULL    |                |
| required      | tinyint(4)   | YES  |     | NULL    |                |
| tablename     | varchar(255) | YES  |     | NULL    |                |
+---------------+--------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)

MariaDB [phplist_db]> select * from phplist_user_attribute;
+----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+
| id | name                                                                                                                          | type     | listorder | default_value | required | tablename      |
+----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+
|  4 | I agree to the OSE <a href=\'https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy\'>Privacy Policy</a>. | checkbox |         0 |               |        1 | termsofservice |
+----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+
1 row in set (0.00 sec)

MariaDB [phplist_db]>

note there were two other tables with 'attribute' in their name, which appear to track & store the fact that each user has accepted the terms of service as required by gpdr

MariaDB [phplist_db]> select * from phplist_admin_attribute
	-> ;
Empty set (0.00 sec)

MariaDB [phplist_db]> select * from phplist_admin_attribute;
Empty set (0.00 sec)

MariaDB [phplist_db]> select * from phplist_user_user_attribute;
+-------------+--------+-------+
| attributeid | userid | value |
+-------------+--------+-------+
|           4 |      3 | on    |
|           4 |      6 | on    |
|           4 |      7 | on    |
|           4 |      8 | on    |
+-------------+--------+-------+
4 rows in set (0.00 sec)

MariaDB [phplist_db]>

I submitted a bug report with all these details to the phplist mantis https://mantis.phplist.org/view.php?id=19503
further testing should be done, modifying the script to ensure that it always populates the $required attributes array and checks to see if theyr'e missing

Mon Oct 29, 2018

revisiting my ajax solution for phplist, I have 3 outstanding issues
1. I need to harden the ACCESS_CONTROL_ALLOW_ORIGIN options and figure out how to set it up for mulitple domains
2. I need to format it better since I removed the tables
3. I need to get it to handle errors properly
the last one, error catching, was discussed here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491
the full code of the ajax form for error handling is given here https://github.com/scotch-io/ajax-forms-jquery/blob/master/magic.js
an explination is found here https://scotch.io/tutorials/submitting-ajax-forms-with-jquery
I tried to get some specific info, but all I got was a status code of 0 and a status text of 'error' — both when I give it proper & improper data.

Fri Oct 26, 2018

I made some progress on the ajax form. did some cleanup & better functionality when JS is disabled

user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html                              100% 5009    49.5KB/s   00:00    
user@ose:~$ cat phplistAjax.html 
<html><body>

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>

<script type="text/javascript">
function checkform()
{
  for (i=0;i<fieldstocheck.length;i++) {
	if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
	  if (document.subscribeform.elements[fieldstocheck[i]].checked) {
	  } else {
		alert("The following field is required:  "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	} else {
	  if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
		alert("Please enter your "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	}
  }
  for (i=0;i<groupstocheck.length;i++) {
	if (!checkGroup(groupstocheck[i],groupnames[i])) {
	  return false;
	}
  }
  
  if (! checkEmail()) {
	alert("Email address is not valid");

	return false;
  }

  return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
  fieldstocheck[fieldstocheck.length] = value;
  fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
  groupstocheck[groupstocheck.length] = value;
  groupnames[groupnames.length] = name;
}

function compareEmail()
{
  return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}

function checkEmail()
{
  var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.subscribeform.elements["email"].value);
}

function checkGroup(name,value)
{
  option = -1;
  for (i=0;i<document.subscribeform.elements[name].length;i++) {
	if (document.subscribeform.elements[name][i].checked) {
	  option = i;
	}
  }
  if (option == -1) {
	alert ("Please enter your "+value);

	return false;
  }

  return true;
}

</script>
<script type="text/javascript">
function submitForm() {
	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#subscribeform').serialize();
	jQuery.ajax( {
		type: 'POST',
		data: data,
		async: true,
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
		dataType: 'html',
		//contentType: 'multipart/form-data',
		success: function (data, status, request) {
			jQuery("#result").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');
		},
		error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
	});
}

</script>

<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">

	<form method="post" action="" name="subscribeform" id="subscribeform">

		<div class="required"><label for="email">Email address *</label></div>
			<input type=text name=email required="required" placeholder="" size="40" id="email" />

		<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
		<input type="hidden" name="htmlemail" value="1" />

		<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
		<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span>
		<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script>

		<input type="hidden" name="list[2]" value="signup" />
		<input type="hidden" name="listname[2]" value="newsletter"/>
		<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>

	</form>

	<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>

</div>

<script type="text/javascript">

// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';

</script>


</body></html>
user@ose:~$

meeting with Marcin about phplist
1. I explained GDPR and how Alex's site is not properly meeting our obligations
2. I asked how we should integrate our phplist newsletter to social media, and Marcin said don't worry about it; we should just delete the links to like/retweet the newsletter posting for now. Perhaps in the future we can script a tool to automatically send a campaign when a new post is made to our wordpress site, then post to social media & add-back the "like" and "retweet" links then. But, for now, we just need to get him suited to send newsletters without jumping through 4 distinct gmail accounts.
3. I asked about attributes. He said that we should import all existing lists' attributes into phplist. therefore, we're going to want to enumerate all of this info into our gdpr-compliant Privacy Policy
4. I explained how phplist ajax & rest api works (or doesn't work) to add users to our subscription list without requring them to leave out other domain. I'm currently ironing-out final issues in the ajax solution.
5. I explained how the repermission campaign would be requried after imports.
6. we discussed how we would have multiple lists one for each of the existing lists, and then we would create one more list for the general "ose newsletter" which only needs the 3 basic & required attributes = timestamp, email address, & accepted PP.

Wed Oct 24, 2018

continuing with my efforts to get a custom form to add a subscriber to our phplist site using AJAX..
the source of offending modsecurity rule id=960010 is in /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf

[root@hetzner2 ~]# grep '960010' -B 30 -A 2 /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
# Restrict which content-types we accept.
#
# TODO Most applications support only two types for request bodies
#      because that is all browsers know how to produce. If you are using
#      automated tools to talk to the application you may be using other
#      content types and would want to change the list of supported types.
# 
#      Note though that ModSecurity parses only three content types:
#      application/x-www-form-urlencoded, multipart/form-data request and 
#      text/xml. The protection provided for any other type is inferior.
#
# TODO There are many applications that are not using multipart/form-data
#      types (typically only used for file uploads). This content type
#      can be disabled if not used.  
#
# NOTE We allow any content type to be specified with GET or HEAD
#      because some tools incorrectly supply content type information
#      even when the body is not present. There is a rule further in
#      the file to prevent GET and HEAD requests to have bodies to we're
#      safe in that respect.
#
# NOTE Use of WebDAV requires "text/xml" content type.
#
# NOTE Philippe Bourcier (pbourcier AT citali DOT com) reports
#      applications running on the PocketPC and AvantGo platforms use
#      non-standard content types:
#
#      M-Business iAnywhere      application/x-mal-client-data
#      UltraLite iAnywhere       application/octet-stream
#
SecRule REQUEST_METHOD "!^(?:GET|HEAD|PROPFIND|OPTIONS)$" "phase:1,chain,t:none,block,msg:'Request content type is not allowed by policy',rev:'2',ver:'OWASP_CRS/2.2.9',maturity:'9',accuracy:'9',id:'960010',tag:'OWASP_CRS/POLICY/ENCODING_NOT_ALLOWED',tag:'WASCTC/WASC-20',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/EE2',tag:'PCI/12.1',severity:'2',logdata:'%{matched_var}'" 
		SecRule REQUEST_HEADERS:Content-Type "^([^;\s]+)" "chain,capture"
				SecRule TX:0 "!^%{tx.allowed_request_content_type}$" "t:none,ctl:forceRequestBodyVariable=On,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-OWASP_CRS/POLICY/CONTENT_TYPE_NOT_ALLOWED-%{matched_var_name}=%{matched_var}"
[root@hetzner2 ~]#

now, I am setting the enctype of the form to "multipart/form-data" in the ajax form, but modsecurity is complaining that it's "text/plain"

<form method="post" action="" name="subscribeform" enctype="multipart/form-data">

I found a list of supported data types in /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf

[root@hetzner2 ~]# grep 'application/xml' /etc/httpd/modsecurity.d/*
grep: /etc/httpd/modsecurity.d/activated_rules: Is a directory
/etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf:  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json', \
[root@hetzner2 ~]#

I also found a note that the offending data type (text/plain) was indeed added to the acceptable list in the CRS 3.0 back in 2016; not sure why we don't have that https://github.com/SpiderLabs/owasp-modsecurity-crs/issues/208
so I could update the CRS's config to include "text/plain", but I think the best solution is to fix the AJAX code to send the correct data type.
the offending code is probably somewhere in this jquery.ajax() call (as obtained from the example zip in the phplist forum thread https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip

function submitForm() {                                                                                                           
	successMessage = 'Thank you for your registration. Please check your email to confirm.';                                      
	data = jQuery('#subscribeform').serialize();                                                                                  
	jQuery.ajax( {                                                                                                                
		type: 'POST',                                                                                                             
		data: data,                                                                                                               
		url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',                                                     
		dataType: 'html',                                                                                                         
		success: function (data, status, request) {                                                                               
			jQuery("#result").empty().append(data != '' ? data : successMessage);                                                 
			jQuery('#attribute1').val('');                                                                                        
			jQuery('#email').val('');                                                                                             
		},                                                                                                                        
		error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }                
	});                                                                                                                           
}

I've never used jquery, but my guess is that we need to set the "dataType" here; it's documented here https://api.jquery.com/jQuery.ajax/
oh, hmm, actualy reading the docs suggest that the 'dataType' defines what data type we expect _back_ from the server https://api.jquery.com/jQuery.ajax/

The type of data that you're expecting back from the server. If none is specified, jQuery will try to infer it based on the MIME type of the response (an XML MIME type will yield XML, in 1.4 JSON will yield a JavaScript object, in 1.4 script will execute the script, and anything else will be returned as a string). The available types (and the result passed as the first argument to your success callback) are:

so I guess I need to dig into the jquery.serialize() command

<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
...
function submitForm() {                                                                                                           
	successMessage = 'Thank you for your registration. Please check your email to confirm.';                                      
	data = jQuery('#subscribeform').serialize(); 
...
}

so the documentation on the serialize function explicitly states that it creates a string https://api.jquery.com/serialize/
ah, so the option I want is "contentType" https://api.jquery.com/jQuery.ajax/

function submitForm() {                                                                               
	successMessage = 'Thank you for your registration. Please check your email to confirm.';          
	data = jQuery('#subscribeform').serialize();                                                          jQuery.ajax( {                                                                                    
		type: 'POST',                                                                                 
		data: data,                                                                                   
		url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',                         
		dataType: 'html',                                                                             
		contentType: 'multipart/form-data',                                                           
		success: function (data, status, request) {                                                   
			jQuery("#result").empty().append(data != '' ? data : successMessage);                     
			jQuery('#attribute1').val('');                                                            
			jQuery('#email').val('');                                                                 
		},                                                                                                    error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }                                                                                          
	});                                                                                               
}

there's a new modsecurity alert

[Wed Oct 24 20:43:47.876714 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Multipart parsing error (init): Multipart: Boundary not found in C-T. [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]
[Wed Oct 24 20:43:47.876765 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "REQBODY_ERROR" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "151"] [id "960912"] [rev "1"] [msg "Failed to parse request body."] [data "Multipart: Boundary not found in C-T."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]

so this is probably because the content is--in fact--actually "text/plain" there's no multipart boundray since I just lied about what the mime type is. The solution should still be to get serialize() to output the proper mime type
well, it appears that the alternative solution (as is often used for form data including binary content--such as images), is to use the FormData, but this is a browser-specific thing https://developer.mozilla.org/en-US/docs/Web/API/FormData
so proceeding with FormData would likely limit the browsers in which this ajax form will work. It's really not requred for the small fields needed for subscriber attributes data in this form. so let's just ignore the modsecurity role
well, I tried to overwrite the existing allowable mime types in our vhost file (making restore from backups easier). that didn't work
I tried to just add the rule id = 960010 to the long list of SecRuleRemoveById list in the vhost file; that also didn't work
the only way it worked as if I actually edited the list in the /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf file, adding "text/plain" to the list (followed by an apache reload) https://stackoverflow.com/questions/41604674/customizing-apache-mod-security-to-accept-content-type-text-plain

[root@hetzner2 modsecurity.d]# pwd
/etc/httpd/modsecurity.d
[root@hetzner2 modsecurity.d]# grep -C5 'text/plain' modsecurity_crs_10_config.conf
SecAction \
  "id:'900012', \
  phase:1, \
  t:none, \
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS', \
  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json|text/plain', \
  setvar:'tx.allowed_http_versions=HTTP/0.9 HTTP/1.0 HTTP/1.1', \
  setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/', \
  setvar:'tx.restricted_headers=/Proxy-Connection/ /Lock-Token/ /Content-Range/ /Translate/ /via/ /if/', \
  nolog, \
  pass"
[root@hetzner2 modsecurity.d]#

so this time it just got a 200 from the server (logs all look clean), but the user didn't get added. The js console shows some (unrelated?) errors about our jquery script file

Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More]
jquery.min.js:112:77
SyntaxError: missing ) after argument list[Learn More]
phplistAjax.php:1:58
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
phplistAjax.php:7:2
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2
ready
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371

I checked out the sourcecode of our phplist subscribe page, and I discovered that it has its own checkform() js function, and some helper functions. I just copied these to my ajax form https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
unrealted, but I saw that at the top of this sourcecode is a php command that for some reason came out as text, not processed by php

<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phpList version 3.3.3" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" href="./images/phplist.ico" />
<title>Subscribe to our Newsletters</title><meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<meta name="theme-color" content="#2C2C2C"/>
<link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" />
<link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" />
<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />
</head>

it looks like this comes from pagetop_minified.php

[root@hetzner2 lists]# pwd
/var/www/html/phplist.opensourceecology.org/public_html/lists
[root@hetzner2 lists]# cat admin/ui/dressprow/pagetop_minified.php 
<?php
/*
  We request you retain the full headers below including the links.
  This not only gives respect to the large amount of time given freely
  by the developers, but also helps build interest, traffic and use of
  phpList, which is beneficial to it's future development.

  Michiel Dethmers, phpList Ltd 2003 - 2015
*/
?>
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="<?php echo $_SESSION['adminlanguage']['iso']?>" lang="<?php echo $_SESSION['adminlanguage']['iso']?>" dir="<?php echo $_SESSION['adminlanguage']['dir']?>">
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phplist version <?php echo VERSION?>" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" id="favicon" href="./images/phplist.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<link rel="stylesheet" href="ui/dressprow/css/all.min.css?v=<?php echo filemtime(dirname(FILE).'/css/all.min.css'); ?>" />

<?php
if (isset($GLOBALS['config']['head'])) {
	foreach ($GLOBALS['config']['head'] as $sHtml) {
		echo $sHtml;
		echo "\n";
		echo "\n";
	}
}
[root@hetzner2 lists]#

interstingly, there's php lines above it (ie: the "phplist version") execute without issues
this is somewhat related, but not really https://mantis.phplist.org/view.php?id=18447
ah, this is it https://mantis.phplist.org/view.php?id=19307
1. looks like the fix was to just remove it https://github.com/phpList/phplist-ui-bootlist/pull/62
anyway, it looks like the error "SyntaxError: missing ) after argument list" was caused by my attribute with a quote in it. Xheni warned me about this here https://mantis.phplist.org/view.php?id=19436
I replaced my links' encapsulating double qoutes with single quotes as a temp fix
ok, better fix, update the attribute to use single quotes in phplist's admin ui
1. so actually the issue is quotes, not slashes.
ok, that elminated the error entirely. Now I just have

Loading mixed (insecure) display content “http://powered.phplist.com/images/3.3.3/power-phplist.png” on a secure page[Learn More]
jquery.min.js:112:77
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]

this is frustrating, but I confirmed that signup *does* work from our actual subscribe page https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1
I noticed that there were a few js files included in the phplist site's code, so I added these to my ajax form

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>                                                                 
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>                                                                                               
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

now I get a pop-up alert = "Sorry, we were unable to process your subscription."
someone else posted about this issue a couple years ago https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
ah, another point. I have the URL wrong! per https://discuss.phplist.org/t/ajax-subscribe-api/974/15
1. I had been using https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1

url: 'https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1',

1. but, in fact, it should use 'asubscribe' intead of 'subscribe' = https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=1
per the thread above, I also changed it to default to html & not confirm email address
it's still not working. this is taking for fucking ever. I gotta get some POC. I'll simplify it by stripping out the "I agree" checkbox for now. If I can fucking get that to work, I'll try to add a simple checkbox and work from there..
I created a new list and cooresponding subscribe page that's as simple as possible: it's just an email address https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=2
ugh, even that won't work.
oh, I got it to work! I had to set both the name and the id attributes of the form field to "subscribeform". Note that the form coming from phplist's html only sets the 'name' attribute, but the jquery serialize() function is called against the 'id' attribute. otherwise, it probably just sent an empty string to the subscribe page!

<form method="post" action="" name="subscribeform" id="subscribeform">

woohoo! I switched back to the list=1 (with the checkbox agreeing to our PP), and it worked! When the checkbox is *not* ticked, a JS pop-up yells at you:

The following field is required: I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.

1. unfortunately, it still says that it fails even when it succeeds, as the thread above pointed out here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
fwiw, this is what woked today. Tomorrow, I should look into making this visible only if JS is enabled. Otherwise, I should just display an anchor link the user to our phplist subscription page

user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html                              100% 5329    53.4KB/s   00:00    
user@ose:~$ cat phplistAjax.html 
<html><body>

<head>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<script type="text/javascript">

function checkform()
{
  for (i=0;i<fieldstocheck.length;i++) {
	if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
	  if (document.subscribeform.elements[fieldstocheck[i]].checked) {
	  } else {
		alert("The following field is required:  "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	} else {
	  if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
		alert("Please enter your "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	}
  }
  for (i=0;i<groupstocheck.length;i++) {
	if (!checkGroup(groupstocheck[i],groupnames[i])) {
	  return false;
	}
  }
  
  if (! checkEmail()) {
	alert("Email address is not valid");

	return false;
  }

  return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
  fieldstocheck[fieldstocheck.length] = value;
  fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
  groupstocheck[groupstocheck.length] = value;
  groupnames[groupnames.length] = name;
}

function compareEmail()
{
  return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}

function checkEmail()
{
  var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.subscribeform.elements["email"].value);
}

function checkGroup(name,value)
{
  option = -1;
  for (i=0;i<document.subscribeform.elements[name].length;i++) {
	if (document.subscribeform.elements[name][i].checked) {
	  option = i;
	}
  }
  if (option == -1) {
	alert ("Please enter your "+value);

	return false;
  }

  return true;
}

</script>
<script type="text/javascript">
function submitForm() {
	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#subscribeform').serialize();
	jQuery.ajax( {
		type: 'POST',
		data: data,
		async: true,
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
		dataType: 'html',
		//contentType: 'multipart/form-data',
		success: function (data, status, request) {
			jQuery("#result").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');
		},
		error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
	});
}

</script>
</head>

<form method="post" action="" name="subscribeform" id="subscribeform"><table border=0>
  <tr><td><div class="required"><label for="email">Email address *</label></div></td>
  <td class="attributeinput"><input type=text name=email required="required" placeholder="" size="40" id="email" />
  <script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script></td></tr><input type="hidden" name="htmlemail" value="1" />
<tr><td colspan="2">
<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span><script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script></td></tr>
</table><input type="hidden" name="list[2]" value="signup" /><input type="hidden" name="listname[2]" value="newsletter"/><div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div><input type=submit name="subscribe" value="Subscribe to the selected newsletters" onClick="return checkform();">    <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website" ><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0" /></a></p></div>

<button class='button' 
	onclick="if (checkform()) {submitForm();} return false;"
>Subscribe</button>
<div id="result" style="color: red;"></div>

</div>



</body></html>
user@ose:~$

Tue Oct 23, 2018

Sam got back to me about integrating a signup list into wordpress via AJAX or the REST API. They said the API doesn't yet support attributes (that conflicts with the example I saw yesterday, though). Sam said adding that feature to the API will come with time, but we could fund it to be built in 6-8 weeks for ~$4,500. Of course, we could also just develop it ourselves and PR it into their repo (it uses Symfony and Doctrine) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/4
1. Sam didn't give a direct answer, but suggested that it's probably possible to do with ajax

fuck it; let's see if I can manually create a form that does what I want with ajax. I'll use this old example as a template https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
I put the file up on the microfactory site in the wp-content dir.
I got a success message (Thank you for your registration. Please check your email to confirm.), but it didn't add to the subscribers list. I checked the html console, and I got an error due to Cross-Origin

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/?p=subscribe&id=1. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘http://phplist.opensourceecology.org’).

I found that phplist inclues logic to define the "Acces-Control-Allow-Origin" header based on a config file option

[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'Allow-Origin' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'ACCESS_CONTROL_ALLOW_ORIGIN' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
public_html/lists/admin/init.php:if (!defined('ACCESS_CONTROL_ALLOW_ORIGIN')) {
public_html/lists/admin/init.php:    define('ACCESS_CONTROL_ALLOW_ORIGIN', $GLOBALS['scheme'].'://'.$_SERVER['HTTP_HOST']);
[root@hetzner2 phplist.opensourceecology.org]#

and here's the relevant option in the config file https://resources.phplist.com/system/config/access_control_allow_origin
apparently the protcol (https://) must be specified. I finally got a new error after adding this line to the phplist config.php file

// allow AJAX queries to add subscribers to our db from other domains                                                      
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );

now the error is

Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More]
jquery.min.js:112:77
SyntaxError: missing ) after argument list[Learn More]
phplistAjax.php:1:58
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
phplistAjax.php:7:2
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2
ready
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange

this time jquery just fucking fetched the whole form and put it on the page somehow.

<div class="panel">
<br>
<div class="content  well"><h3>Subscribe to our newsletters</h3>

<div class="error"><span class="required">* required fields are marked red</span></div>
<div class="error missing">The following required values are missing: Email addresses you entered do not match</div><br>

<form method="post" action="" name="subscribeform"><table border="0">
  <tbody><tr><td><div class="required"><label for="email">Email address *</label></div></td>
  <td class="attributeinput"><input name="email" required="required" placeholder="testa@example.com" size="40" id="email" type="text">
  </td></tr>
  <tr><td><div class="required"><label for="confirm">Confirm your email address *</label></div></td>
  <td class="attributeinput"><input name="emailconfirm" required="required" value="" size="40" id="confirm" type="text">
  </td></tr><tr><td colspan="2">
	  <span class="attributeinput">
	  <input name="textemail" value="1" id="textemail" type="checkbox"></span>
	  <span class="attributename"><label for="textemail">I prefer to receive emails in Text format</label></span>
	  </td></tr>
<tr><td colspan="2">
<input name="attribute4" value="on" class="attributeinput" id="attribute4" type="checkbox">
<span class="required"><label for="attribute4">I agree to the OSE <a href="https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy">Privacy Policy</a>. *</label></span></td></tr>
</tbody></table><input name="list[2]" value="signup" type="hidden"><input name="listname[2]" value="newsletter" type="hidden"><div style="display:none"><input name="VerificationCodeX" value="" size="20" type="text"></div><input name="subscribe" value="Subscribe to the selected newsletters" onclick="return checkform();" type="submit">    <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website"><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0"></a></p></div>
</div>

I used this html form data to merge into the example one
I hit a modsecurity issue with rule #960010

Request content type is not allowed by policy

1. I think it may be related to the content type submitted by the form somewhere defaulting to "text/plain" *shrug*

I had lost my password for our phplist install due to my ssd crash; I reset it (noting that you cannot login with your email address; you must use your username--as addressed in the password reset email. In my case, 'maltfield').
I also added an admin for marcin
I confirmed that the shared 'admin' user is in our shared ose keepass db

Mon Oct 22, 2018

Marcin sent me our other 3 email spreadsheets; I dug through them to see what attributes we collected
according to phplist plugin page, the rest api doesn't support setting attributes https://resources.phplist.com/plugin/restapi
1. however, there does appear to be support for non-email attributes per this document, even though it does mention on the main page of this github repo that it's deprecated (the replacement for php4 isn't stable, however) https://github.com/phpList/phplist-plugin-restapi/blob/master/plugins/restapi2/docs/api.apib#L185-189
the replacement api's repo does say that it can work for php3 https://github.com/phpList/rest-api

This new REST API can also be used to provide REST access to an existing phpList 3 installation. For this, the phpList 3 installation and the phpList 4 installation with the REST API need to share the same database. For security reasons, the REST APIs from phpList 3 and phpList 4 should not be used for the same database in parallel, though.

Sat Oct 20, 2018

I did a search for wordpress plugins using the keyword "phplist". Here's some noteable results
1. Sign Me Up https://wordpress.org/plugins/sign-me-up/
  1. - last updated 3 years ago
  2. + explicitly lists that it connects with phplist using AJAX
  3. - says that it doesn't support additional attributes (email address only). This will probably not work as we need to at least have a checkbox for the user to accept our gdpr-ready Privacy Policy
  4. there's a lot of info about this plugin here https://www.jaromy.net/wordpress-plugins/sign-me-up[[1]]
2. WP PHPList https://wordpress.org/plugins/phplist-form-integration/
  1. - last updated 9 years ago
3. yPHPlista https://wordpress.org/plugins/yphplista/
  1. - last updated 8 years ago https://wordpress.org/plugins/yphplista/

I tried searching the phplist forums for "wordpress" to dig through recent discussions about integrating phplist with wordpress
1. I found this interesting discussion hinting to having a phplist plugin automatically send email campaigns after a new wordpress post is in. I really like this idea (ie: RSS scraping -> email campaign). Prior to sending the email campaign, we could post a link to it on facebook, twitter, etc. Then we could add a link to "like" or "retweet" the post (on social media) to the original post (on wordpress) within the email template. https://discuss.phplist.org/t/how-is-content-generated-in-wordpress/1656/6
2. so 6 months ago someone asked my question: is there an official wp plugin for phplist? Sam Tuke (CEO of phplist) responded stating that there isn't but it's needed. And expounded on how the AJAX solution is the old way and that the future phplist wp plugin would work via the REST API (I like this much better too) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894
3. this may work with the AJAX way https://discuss.phplist.org/t/wordpress-integration-newsletter-sign-up-plugin/1723/15
4. And there was this old post for 2016 about the 9-year-old "WP PHPList" (phplist-form-integration) wordpress plugin https://discuss.phplist.org/t/phplist-wordpress-integration-solved/2279/2

I lost my password for phplist's discourse site due to my ssd crash, so I generated a new one
Interestingly, it looks like phplist's discourse site is SSO using wordpress as the account backend
I replied to this post, asking Sam if there's an option for sending our new subscriber's attributes to the stable version phplist via either AJAX or REST https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/3

Fri Oct 19, 2018

I'm back in NYC after restoring my laptop following my ssd crash
sent Sara information about wp-cli and managing wordpress
1. I also sent an email to Marcin asking if he authorizes them having ssh and/or shraed ose keepass access
2. specifically, the plugin that Sara wanted to install was this https://wordpress.org/plugins/custom-css-js
3. the above plugin is freemium. the reviews suggested this free alterantive for more features, but it's a less popular plugin https://wordpress.org/plugins/scripts-n-styles/
Marcin sent me a confusing email about utilization of google analytics. One of our first conversations with Marcin back in 2017 was about how OSE was working to migrate _away_ from Google services. We've since stripped analytics from all our sites. We looked at Piwik, but--due to security issues--we decided to go with awstats.
1. I restated that we use awstats, and asked what are the benefits of using google analytics
2. I added the microfactory site to our awstast generation cron job, and confirmed that I could view all the stats so far this month https://awstats.opensourceecology.org:4443/awstats.microfactory.opensourceecology.org.html
3. In general, I'm very confused about where the line is drawn between giving up on FLOSS and moving to some closed-source SaaS solution here at OSE.
  1. I've been hacking the hell out of phplist so that we can use an ugly FLOSS solution, but Mailchimp would be much better, and very cheap for our needs.
  2. We still don't have a decent issue tracker. Mantis is a long-standing TODO, but we could use Jira for free because we're a nonprofit.
  3. Google Analytics is surely more powerful thatn awstats, but does that really justify the use of closed-source tools that exploit our visitor's data? Why draw the line here between awstats & GA but not use Jira or Mailchimp like most other nonprofits?
Marcin sent me an error when trying to post our email thread about installing a wordpress plugin to the wiki
1. fixed by modsec whitelist
  1. 950019, generic attack = email injection
I updated the documentation on the wiki for the quick-start guide to configuring 2FA for our wordpress sites using the Google Authenticator plugin https://wiki.opensourceecology.org/wiki/2FA#Quick_Start_Guide
1. I added a step to enable relaxed mode, so the user's phone only has to be correct within 4 minutes, rather than 30 seconds.
2. I recommended use of the oandbackup app, which is the FLOSS replacement for Titanium Backup https://f-droid.org/en/packages/dk.jens.backup/

...

began to revisit my install of the "Newsletter Sign-Up" wordpress plugin on the fef site
I logged into the fef wordpress ite
I navigated to the Plugins -> "Newsletter Sign-Up" -> "Settings" link (note that this redirects to the main panel nav bar's "Newsl. Sign-Up" link.
1. I changed "Select your mailinglist provider" to "PHPList"
2. our subscriber link is this, so I left the "PHPList list ID" at "1" https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
3. I set the "Newsletter form action" to "https://phplist.opensourceecology.org/lists/?p=subscribe&id=1"
this plugin appears to be mostly built around the trick of having someone signup when leaving a comment. That's a feature that we may want to disable. Mostly we want a widget and a form embedded into a post/page.
1. I picked a bad site, fef, for testing widgets. it doesn't use them!
2. I tested adding a form for signing up to the phplist newsletter to the lola page by adding "[nsu_form]" to the content of the post https://fef.opensourceecology.org/2015/09/03/lola/
  1. when I submitted on this form, it just fucking redirected me to the phplist site. it didn't even auto-fill the form! what rubbish! what we want here is for a seamless submittion to the phplist site without the user having to leave the site that they are on.
ugh, I just realized that the "Newsletter Sign-Up" form's main plugin page on the wp site shows a message indicating that the plugin is no longer maintained (it was last updated 11 months ago) https://wordpress.org/plugins/newsletter-sign-up/
I deactivated the "Newsletter Sign-Up" plugin and deleted the related content & widgets from the fef site.
looks like I lost my logs from when I discovered the "Newsletter Sign-Up" and similar wordpress plugins
I dug up some options
1. https://wordpress.org/plugins/sign-me-up/
  1. - last updated 3 years ago
2. https://wordpress.org/plugins/yphplista/
  1. - last updated 8 years ago
it looks like phplist.com (not .org) offers a solution to this issue, the so-called "AJAX" solution so subscribres can signup "...and stay on your website" https://www.phplist.com/ajaxdemo
1. found some info on enabling this the '.org' way. note that there may be issues with the ACCESS_CONTROL_ALLOW_ORIGIN https://discuss.phplist.org/t/ajax-subscribe-api/974/7

define('ACCESS_CONTROL_ALLOW_ORIGIN', 'www.mydomain.de');
define('PUBLIC_PROTOCOL','https');

I found a few resources about ajax on phplist.com's wiki https://resources.phplist.com/start?do=search&id=ajax
1. ugh, this just references the 2x no-longer-maintained plugins I found above https://resources.phplist.com/develop/wordpress?s[]=ajax
there's only 1x reference to "ajax" in the phplist.org manual it referes to the Ethical Pet case study. Yeah, the one that migrated off from phplist..
important, but a fucking stub. can we get an example listing 3x domains? https://resources.phplist.com/system/config/access_control_allow_origin?s[]=ajax
the discuss thread above includes this example file from subscription_form_example.html (in subscription_form_example.html.zip) https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
1. ugh, the first line includes the jquery src @ googleapis.com. Let's not make our users require decentraleyes to prevent google from tracking them on our site..

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>

it's looking like the best option is to use the no longer maintained "Newsletter Sign-Up" wp plugin still :\ I re-enabled it and added the line "[nsu_form]" to the lola paga again. I'll try to debug that before reinventing the wheel https://fef.opensourceecology.org/2015/09/03/lola/
1. I checked the code; there's not even any JS; it's just a fucking form with submit action to our distinct phplist site (I figured it was some ajax cross-site rejection failure causing fall-back to the redirect). Maybe this plugin isn't what we want still..I disabled it again.

Sun Oct 14, 2018

bought & installed my replacement ssd after my old one died unexpectedly
install os, restore backups, etc
catching up on old email

Mon Oct 01, 2018

Note that my logs for the first few weeks of this quarter were lost when my laptop's ssd crashed. Please mind the gap.

Maltfield Log/2018 Q4: Difference between revisions

Revision as of 12:07, 3 December 2018

Contents

See Also

Tue Nov 29, 2018

Wed Nov 28, 2018

Tue Nov 27, 2018

Mon Nov 26, 2018

Sat Nov 24, 2018

Fri Nov 23, 2018

Thr Nov 22, 2018

Sun Nov 07, 2018

Sun Nov 04, 2018

Fri Nov 02, 2018

Tue Oct 30, 2018

Mon Oct 29, 2018

Fri Oct 26, 2018

Wed Oct 24, 2018

Tue Oct 23, 2018

Mon Oct 22, 2018

Sat Oct 20, 2018

Fri Oct 19, 2018

Sun Oct 14, 2018

Mon Oct 01, 2018

Navigation menu

Maltfield Log/2018 Q4: Difference between revisions

Revision as of 12:07, 3 December 2018

See Also

Tue Nov 29, 2018

Wed Nov 28, 2018

Tue Nov 27, 2018

Mon Nov 26, 2018

Sat Nov 24, 2018

Fri Nov 23, 2018

Thr Nov 22, 2018

Sun Nov 07, 2018

Sun Nov 04, 2018

Fri Nov 02, 2018

Tue Oct 30, 2018

Mon Oct 29, 2018

Fri Oct 26, 2018

Wed Oct 24, 2018

Tue Oct 23, 2018

Mon Oct 22, 2018

Sat Oct 20, 2018

Fri Oct 19, 2018

Sun Oct 14, 2018

Mon Oct 01, 2018

Navigation menu

Search