Maltfield Log/2018 Q4: Difference between revisions

From Open Source Ecology
Jump to navigation Jump to search
No edit summary
No edit summary
Line 7: Line 7:
# [[User:Maltfield]]
# [[User:Maltfield]]
# [[Special:Contributions/Maltfield]]
# [[Special:Contributions/Maltfield]]
=Fri Oct 26, 2018=
# I made some progress on the ajax form. did some cleanup & better functionality when JS is disabled
<pre>
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html                              100% 5009    49.5KB/s  00:00   
user@ose:~$ cat phplistAjax.html
<html><body>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>
<script type="text/javascript">
function checkform()
{
  for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
  if (document.subscribeform.elements[fieldstocheck[i]].checked) {
  } else {
alert("The following field is required:  "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
  }
} else {
  if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
  }
}
  }
  for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
  return false;
}
  }
 
  if (! checkEmail()) {
alert("Email address is not valid");
return false;
  }
  return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
  fieldstocheck[fieldstocheck.length] = value;
  fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
  groupstocheck[groupstocheck.length] = value;
  groupnames[groupnames.length] = name;
}
function compareEmail()
{
  return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}
function checkEmail()
{
  var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.subscribeform.elements["email"].value);
}
function checkGroup(name,value)
{
  option = -1;
  for (i=0;i<document.subscribeform.elements[name].length;i++) {
if (document.subscribeform.elements[name][i].checked) {
  option = i;
}
  }
  if (option == -1) {
alert ("Please enter your "+value);
return false;
  }
  return true;
}
</script>
<script type="text/javascript">
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
async: true,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
//contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
</script>
<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">
<form method="post" action="" name="subscribeform" id="subscribeform">
<div class="required"><label for="email">Email address *</label></div>
<input type=text name=email required="required" placeholder="" size="40" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
<input type="hidden" name="htmlemail" value="1" />
<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span>
<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script>
<input type="hidden" name="list[2]" value="signup" />
<input type="hidden" name="listname[2]" value="newsletter"/>
<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>
</form>
<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>
</div>
<script type="text/javascript">
// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';
</script>
</body></html>
user@ose:~$
</pre>
# meeting with Marcin about phplist
## I explained GDPR and how Alex's site is not properly meeting our obligations
## I asked how we should integrate our phplist newsletter to social media, and Marcin said don't worry about it; we should just delete the links to like/retweet the newsletter posting for now. Perhaps in the future we can script a tool to automatically send a campaign when a new post is made to our wordpress site, then post to social media & add-back the "like" and "retweet" links then. But, for now, we just need to get him suited to send newsletters without jumping through 4 distinct gmail accounts.
## I asked about attributes. He said that we should import all existing lists' attributes into phplist. therefore, we're going to want to enumerate all of this info into our gdpr-compliant Privacy Policy
## I explained how phplist ajax & rest api works (or doesn't work) to add users to our subscription list without requring them to leave out other domain. I'm currently ironing-out final issues in the ajax solution.
## I explained how the repermission campaign would be requried after imports.
## we discussed how we would have multiple lists  one for each of the existing lists, and then we would create one more list for the general "ose newsletter" which only needs the 3 basic & required attributes = timestamp, email address, & accepted PP.
=Wed Oct 24, 2018=
# continuing with my efforts to get a custom form to add a subscriber to our phplist site using AJAX..
# the source of offending modsecurity rule id=960010 is in /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
<pre>
[root@hetzner2 ~]# grep '960010' -B 30 -A 2 /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
# Restrict which content-types we accept.
#
# TODO Most applications support only two types for request bodies
#      because that is all browsers know how to produce. If you are using
#      automated tools to talk to the application you may be using other
#      content types and would want to change the list of supported types.
#
#      Note though that ModSecurity parses only three content types:
#      application/x-www-form-urlencoded, multipart/form-data request and
#      text/xml. The protection provided for any other type is inferior.
#
# TODO There are many applications that are not using multipart/form-data
#      types (typically only used for file uploads). This content type
#      can be disabled if not used. 
#
# NOTE We allow any content type to be specified with GET or HEAD
#      because some tools incorrectly supply content type information
#      even when the body is not present. There is a rule further in
#      the file to prevent GET and HEAD requests to have bodies to we're
#      safe in that respect.
#
# NOTE Use of WebDAV requires "text/xml" content type.
#
# NOTE Philippe Bourcier (pbourcier AT citali DOT com) reports
#      applications running on the PocketPC and AvantGo platforms use
#      non-standard content types:
#
#      M-Business iAnywhere      application/x-mal-client-data
#      UltraLite iAnywhere      application/octet-stream
#
SecRule REQUEST_METHOD "!^(?:GET|HEAD|PROPFIND|OPTIONS)$" "phase:1,chain,t:none,block,msg:'Request content type is not allowed by policy',rev:'2',ver:'OWASP_CRS/2.2.9',maturity:'9',accuracy:'9',id:'960010',tag:'OWASP_CRS/POLICY/ENCODING_NOT_ALLOWED',tag:'WASCTC/WASC-20',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/EE2',tag:'PCI/12.1',severity:'2',logdata:'%{matched_var}'"
SecRule REQUEST_HEADERS:Content-Type "^([^;\s]+)" "chain,capture"
SecRule TX:0 "!^%{tx.allowed_request_content_type}$" "t:none,ctl:forceRequestBodyVariable=On,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-OWASP_CRS/POLICY/CONTENT_TYPE_NOT_ALLOWED-%{matched_var_name}=%{matched_var}"
[root@hetzner2 ~]#
</pre>
# now, I am setting the enctype of the form to "multipart/form-data" in the ajax form, but modsecurity is complaining that it's "text/plain"
<pre>
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
</pre>
# I found a list of supported data types in /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
<pre>
[root@hetzner2 ~]# grep 'application/xml' /etc/httpd/modsecurity.d/*
grep: /etc/httpd/modsecurity.d/activated_rules: Is a directory
/etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf:  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json', \
[root@hetzner2 ~]#
</pre>
# I also found a note that the offending data type (text/plain) was indeed added to the acceptable list in the CRS 3.0 back in 2016; not sure why we don't have that https://github.com/SpiderLabs/owasp-modsecurity-crs/issues/208
# so I could update the CRS's config to include "text/plain", but I think the best solution is to fix the AJAX code to send the correct data type.
# the offending code is probably somewhere in this jquery.ajax() call (as obtained from the example zip in the phplist forum thread https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
<pre>
function submitForm() {                                                                                                         
successMessage = 'Thank you for your registration. Please check your email to confirm.';                                     
data = jQuery('#subscribeform').serialize();                                                                                 
jQuery.ajax( {                                                                                                               
type: 'POST',                                                                                                           
data: data,                                                                                                             
url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',                                                   
dataType: 'html',                                                                                                       
success: function (data, status, request) {                                                                             
jQuery("#result").empty().append(data != '' ? data : successMessage);                                               
jQuery('#attribute1').val('');                                                                                       
jQuery('#email').val('');                                                                                           
},                                                                                                                       
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }               
});                                                                                                                         
}                                                                                                                               
</pre>
# I've never used jquery, but my guess is that we need to set the "dataType" here; it's documented here https://api.jquery.com/jQuery.ajax/
# oh, hmm, actualy reading the docs suggest that the 'dataType' defines what data type we expect _back_ from the server https://api.jquery.com/jQuery.ajax/
<blockquote>
The type of data that you're expecting back from the server. If none is specified, jQuery will try to infer it based on the MIME type of the response (an XML MIME type will yield XML, in 1.4 JSON will yield a JavaScript object, in 1.4 script will execute the script, and anything else will be returned as a string). The available types (and the result passed as the first argument to your success callback) are:
</blockquote>
# so I guess I need to dig into the jquery.serialize() command
<pre>
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
...
function submitForm() {                                                                                                         
successMessage = 'Thank you for your registration. Please check your email to confirm.';                                     
data = jQuery('#subscribeform').serialize();
...
}
</pre>
# so the documentation on the serialize function explicitly states that it creates a string https://api.jquery.com/serialize/
# ah, so the option I want is "contentType" https://api.jquery.com/jQuery.ajax/
<pre>
function submitForm() {                                                                             
successMessage = 'Thank you for your registration. Please check your email to confirm.';         
data = jQuery('#subscribeform').serialize();                                                          jQuery.ajax( {                                                                                   
type: 'POST',                                                                               
data: data,                                                                                 
url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',                       
dataType: 'html',                                                                           
contentType: 'multipart/form-data',                                                         
success: function (data, status, request) {                                                 
jQuery("#result").empty().append(data != '' ? data : successMessage);                   
jQuery('#attribute1').val('');                                                           
jQuery('#email').val('');                                                               
},                                                                                                    error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }                                                                                         
});                                                                                             
}
</pre>
# there's a new modsecurity alert
<pre>
[Wed Oct 24 20:43:47.876714 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Multipart parsing error (init): Multipart: Boundary not found in C-T. [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]
[Wed Oct 24 20:43:47.876765 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "REQBODY_ERROR" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "151"] [id "960912"] [rev "1"] [msg "Failed to parse request body."] [data "Multipart: Boundary not found in C-T."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]
</pre>
# so this is probably because the content is--in fact--actually "text/plain"  there's no multipart boundray since I just lied about what the mime type is. The solution should still be to get serialize() to output the proper mime type
# well, it appears that the alternative solution (as is often used for form data including binary content--such as images), is to use the FormData, but this is a browser-specific thing https://developer.mozilla.org/en-US/docs/Web/API/FormData
# so proceeding with FormData would likely limit the browsers in which this ajax form will work. It's really not requred for the small fields needed for subscriber attributes data in this form. so let's just ignore the modsecurity role
# well, I tried to overwrite the existing allowable mime types in our vhost file (making restore from backups easier). that didn't work
# I tried to just add the rule id = 960010 to the long list of SecRuleRemoveById list in the vhost file; that also didn't work
# the only way it worked as if I actually edited the list in the /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf file, adding "text/plain" to the list (followed by an apache reload) https://stackoverflow.com/questions/41604674/customizing-apache-mod-security-to-accept-content-type-text-plain
<pre>
[root@hetzner2 modsecurity.d]# pwd
/etc/httpd/modsecurity.d
[root@hetzner2 modsecurity.d]# grep -C5 'text/plain' modsecurity_crs_10_config.conf
SecAction \
  "id:'900012', \
  phase:1, \
  t:none, \
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS', \
  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json|text/plain', \
  setvar:'tx.allowed_http_versions=HTTP/0.9 HTTP/1.0 HTTP/1.1', \
  setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/', \
  setvar:'tx.restricted_headers=/Proxy-Connection/ /Lock-Token/ /Content-Range/ /Translate/ /via/ /if/', \
  nolog, \
  pass"
[root@hetzner2 modsecurity.d]#
</pre>
# so this time it just got a 200 from the server (logs all look clean), but the user didn't get added. The js console shows some (unrelated?) errors about our jquery script file
<pre>
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More]
jquery.min.js:112:77
SyntaxError: missing ) after argument list[Learn More]
phplistAjax.php:1:58
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
phplistAjax.php:7:2
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2
ready
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
</pre>
# I checked out the sourcecode of our phplist subscribe page, and I discovered that it has its own checkform() js function, and some helper functions. I just copied these to my ajax form https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
# unrealted, but I saw that at the top of this sourcecode is a php command that for some reason came out as text, not processed by php
<pre>
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phpList version 3.3.3" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" href="./images/phplist.ico" />
<title>Subscribe to our Newsletters</title><meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<meta name="theme-color" content="#2C2C2C"/>
<link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" />
<link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" />
<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />
</head>
</pre>
# it looks like this comes from pagetop_minified.php
<pre>
[root@hetzner2 lists]# pwd
/var/www/html/phplist.opensourceecology.org/public_html/lists
[root@hetzner2 lists]# cat admin/ui/dressprow/pagetop_minified.php
<?php
/*
  We request you retain the full headers below including the links.
  This not only gives respect to the large amount of time given freely
  by the developers, but also helps build interest, traffic and use of
  phpList, which is beneficial to it's future development.
  Michiel Dethmers, phpList Ltd 2003 - 2015
*/
?>
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="<?php echo $_SESSION['adminlanguage']['iso']?>" lang="<?php echo $_SESSION['adminlanguage']['iso']?>" dir="<?php echo $_SESSION['adminlanguage']['dir']?>">
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phplist version <?php echo VERSION?>" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" id="favicon" href="./images/phplist.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<link rel="stylesheet" href="ui/dressprow/css/all.min.css?v=<?php echo filemtime(dirname(FILE).'/css/all.min.css'); ?>" />
<?php
if (isset($GLOBALS['config']['head'])) {
foreach ($GLOBALS['config']['head'] as $sHtml) {
echo $sHtml;
echo "\n";
echo "\n";
}
}
[root@hetzner2 lists]#
</pre>
# interstingly, there's php lines above it (ie: the "phplist version") execute without issues
# this is somewhat related, but not really https://mantis.phplist.org/view.php?id=18447
# ah, this is it https://mantis.phplist.org/view.php?id=19307
## looks like the fix was to just remove it https://github.com/phpList/phplist-ui-bootlist/pull/62
# anyway, it looks like the error "SyntaxError: missing ) after argument list" was caused by my attribute with a quote in it. Xheni warned me about this here https://mantis.phplist.org/view.php?id=19436
# I replaced my links' encapsulating double qoutes with single quotes as a temp fix
# ok, better fix, update the attribute to use single quotes in phplist's admin ui
## so actually the issue is quotes, not slashes.
# ok, that elminated the error entirely. Now I just have
<pre>
Loading mixed (insecure) display content “http://powered.phplist.com/images/3.3.3/power-phplist.png” on a secure page[Learn More]
jquery.min.js:112:77
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
</pre>
# this is frustrating, but I confirmed that signup *does* work from our actual subscribe page https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1
# I noticed that there were a few js files included in the phplist site's code, so I added these to my ajax form
<pre>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>                                                               
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>                                                                                             
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>     
</pre>
# now I get a pop-up alert = "Sorry, we were unable to process your subscription."
# someone else posted about this issue a couple years ago https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
# ah, another point. I have the URL wrong! per https://discuss.phplist.org/t/ajax-subscribe-api/974/15
## I had been using https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1
<pre>
url: 'https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1',
</pre>
## but, in fact, it should use 'asubscribe' intead of 'subscribe' = https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=1
# per the thread above, I also changed it to default to html & not confirm email address
# it's still not working. this is taking for fucking ever. I gotta get some POC. I'll simplify it by stripping out the "I agree" checkbox for now. If I can fucking get that to work, I'll try to add a simple checkbox and work from there..
# I created a new list and cooresponding subscribe page that's as simple as possible: it's just an email address https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=2
# ugh, even that won't work.
# oh, I got it to work! I had to set both the name and the id attributes of the form field to "subscribeform". Note that the form coming from phplist's html only sets the 'name' attribute, but the jquery serialize() function is called against the 'id' attribute. otherwise, it probably just sent an empty string to the subscribe page!
<pre>
<form method="post" action="" name="subscribeform" id="subscribeform">
</pre>
# woohoo! I switched back to the list=1 (with the checkbox agreeing to our PP), and it worked! When the checkbox is *not* ticked, a JS pop-up yells at you:
<blockquote>
The following field is required:  I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.
</blockquote>
## unfortunately, it still says that it fails even when it succeeds, as the thread above pointed out here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
# fwiw, this is what woked today. Tomorrow, I should look into making this visible only if JS is enabled. Otherwise, I should just display an anchor link the user to our phplist subscription page
<pre>
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html                              100% 5329    53.4KB/s  00:00   
user@ose:~$ cat phplistAjax.html
<html><body>
<head>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<script type="text/javascript">
function checkform()
{
  for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
  if (document.subscribeform.elements[fieldstocheck[i]].checked) {
  } else {
alert("The following field is required:  "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
  }
} else {
  if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
  }
}
  }
  for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
  return false;
}
  }
 
  if (! checkEmail()) {
alert("Email address is not valid");
return false;
  }
  return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
  fieldstocheck[fieldstocheck.length] = value;
  fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
  groupstocheck[groupstocheck.length] = value;
  groupnames[groupnames.length] = name;
}
function compareEmail()
{
  return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}
function checkEmail()
{
  var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.subscribeform.elements["email"].value);
}
function checkGroup(name,value)
{
  option = -1;
  for (i=0;i<document.subscribeform.elements[name].length;i++) {
if (document.subscribeform.elements[name][i].checked) {
  option = i;
}
  }
  if (option == -1) {
alert ("Please enter your "+value);
return false;
  }
  return true;
}
</script>
<script type="text/javascript">
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
async: true,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
//contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
</script>
</head>
<form method="post" action="" name="subscribeform" id="subscribeform"><table border=0>
  <tr><td><div class="required"><label for="email">Email address *</label></div></td>
  <td class="attributeinput"><input type=text name=email required="required" placeholder="" size="40" id="email" />
  <script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script></td></tr><input type="hidden" name="htmlemail" value="1" />
<tr><td colspan="2">
<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span><script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script></td></tr>
</table><input type="hidden" name="list[2]" value="signup" /><input type="hidden" name="listname[2]" value="newsletter"/><div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div><input type=submit name="subscribe" value="Subscribe to the selected newsletters" onClick="return checkform();"> &nbsp;&nbsp; <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&amp;utm_medium=poweredhostedimg&amp;utm_campaign=phpList" title="visit the phpList website" ><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, &copy; phpList ltd" alt="powered by phpList 3.3.3, &copy; phpList ltd" border="0" /></a></p></div>
<button class='button'
onclick="if (checkform()) {submitForm();} return false;"
>Subscribe</button>
<div id="result" style="color: red;"></div>
</div>
</body></html>
user@ose:~$
</pre>
=Tue Oct 23, 2018=
# Sam got back to me about integrating a signup list into wordpress via AJAX or the REST API. They said the API doesn't yet support attributes (that conflicts with the example I saw yesterday, though). Sam said adding that feature to the API will come with time, but we could fund it to be built in 6-8 weeks for ~$4,500. Of course, we could also just develop it ourselves and PR it into their repo (it uses Symfony and Doctrine) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/4
## Sam didn't give a direct answer, but suggested that it's probably possible to do with ajax
# fuck it; let's see if I can manually create a form that does what I want with ajax. I'll use this old example as a template https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
# I put the file up on the microfactory site in the wp-content dir.
# I got a success message (Thank you for your registration. Please check your email to confirm.), but it didn't add to the subscribers list. I checked the html console, and I got an error due to Cross-Origin
<pre>
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/?p=subscribe&id=1. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘http://phplist.opensourceecology.org’).
</pre>
# I found that phplist inclues logic to define the "Acces-Control-Allow-Origin" header based on a config file option
<pre>
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'Allow-Origin' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'ACCESS_CONTROL_ALLOW_ORIGIN' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
public_html/lists/admin/init.php:if (!defined('ACCESS_CONTROL_ALLOW_ORIGIN')) {
public_html/lists/admin/init.php:    define('ACCESS_CONTROL_ALLOW_ORIGIN', $GLOBALS['scheme'].'://'.$_SERVER['HTTP_HOST']);
[root@hetzner2 phplist.opensourceecology.org]#
</pre>
# and here's the relevant option in the config file https://resources.phplist.com/system/config/access_control_allow_origin
# apparently the protcol (https://) must be specified. I finally got a new error after adding this line to the phplist config.php file
<pre>
// allow AJAX queries to add subscribers to our db from other domains                                                     
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );
</pre>
# now the error is
<pre>
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More]
jquery.min.js:112:77
SyntaxError: missing ) after argument list[Learn More]
phplistAjax.php:1:58
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
phplistAjax.php:7:2
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2
ready
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
</pre>
# this time jquery just fucking fetched the whole form and put it on the page somehow.
<pre>
<div class="panel">
<br>
<div class="content  well"><h3>Subscribe to our newsletters</h3>
<div class="error"><span class="required">* required fields are marked red</span></div>
<div class="error missing">The following required values are missing: Email addresses you entered do not match</div><br>
<form method="post" action="" name="subscribeform"><table border="0">
  <tbody><tr><td><div class="required"><label for="email">Email address *</label></div></td>
  <td class="attributeinput"><input name="email" required="required" placeholder="testa@example.com" size="40" id="email" type="text">
  </td></tr>
  <tr><td><div class="required"><label for="confirm">Confirm your email address *</label></div></td>
  <td class="attributeinput"><input name="emailconfirm" required="required" value="" size="40" id="confirm" type="text">
  </td></tr><tr><td colspan="2">
  <span class="attributeinput">
  <input name="textemail" value="1" id="textemail" type="checkbox"></span>
  <span class="attributename"><label for="textemail">I prefer to receive emails in Text format</label></span>
  </td></tr>
<tr><td colspan="2">
<input name="attribute4" value="on" class="attributeinput" id="attribute4" type="checkbox">
<span class="required"><label for="attribute4">I agree to the OSE <a href="https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy">Privacy Policy</a>. *</label></span></td></tr>
</tbody></table><input name="list[2]" value="signup" type="hidden"><input name="listname[2]" value="newsletter" type="hidden"><div style="display:none"><input name="VerificationCodeX" value="" size="20" type="text"></div><input name="subscribe" value="Subscribe to the selected newsletters" onclick="return checkform();" type="submit"> &nbsp;&nbsp; <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&amp;id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&amp;utm_medium=poweredhostedimg&amp;utm_campaign=phpList" title="visit the phpList website"><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0"></a></p></div>
</div>
</pre>
# I used this html form data to merge into the example one
# I hit a modsecurity issue with rule #960010
<pre>
Request content type is not allowed by policy
</pre>
## I think it may be related to the content type submitted by the form somewhere defaulting to "text/plain" *shrug*
# I had lost my password for our phplist install due to my ssd crash; I reset it (noting that you cannot login with your email address; you must use your username--as addressed in the password reset email. In my case, 'maltfield').
# I also added an admin for marcin
# I confirmed that the shared 'admin' user is in our shared ose keepass db
=Mon Oct 22, 2018=
# Marcin sent me our other 3 email spreadsheets; I dug through them to see what attributes we collected
# according to phplist plugin page, the rest api doesn't support setting attributes https://resources.phplist.com/plugin/restapi
## however, there does appear to be support for non-email attributes per this document, even though it does mention on the main page of this github repo that it's deprecated (the replacement for php4 isn't stable, however) https://github.com/phpList/phplist-plugin-restapi/blob/master/plugins/restapi2/docs/api.apib#L185-189
# the replacement api's repo does say that it can work for php3 https://github.com/phpList/rest-api
<blockquote>
This new REST API can also be used to provide REST access to an existing phpList 3 installation. For this, the phpList 3 installation and the phpList 4 installation with the REST API need to share the same database. For security reasons, the REST APIs from phpList 3 and phpList 4 should not be used for the same database in parallel, though.
</blockquote>


=Sat Oct 20, 2018=
=Sat Oct 20, 2018=

Revision as of 21:50, 29 October 2018

My work log from the year 2018 Quarter 4. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

  1. Maltfield_Log
  2. User:Maltfield
  3. Special:Contributions/Maltfield

Fri Oct 26, 2018

  1. I made some progress on the ajax form. did some cleanup & better functionality when JS is disabled
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html                              100% 5009    49.5KB/s   00:00    
user@ose:~$ cat phplistAjax.html 
<html><body>

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>

<script type="text/javascript">
function checkform()
{
  for (i=0;i<fieldstocheck.length;i++) {
	if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
	  if (document.subscribeform.elements[fieldstocheck[i]].checked) {
	  } else {
		alert("The following field is required:  "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	} else {
	  if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
		alert("Please enter your "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	}
  }
  for (i=0;i<groupstocheck.length;i++) {
	if (!checkGroup(groupstocheck[i],groupnames[i])) {
	  return false;
	}
  }
  
  if (! checkEmail()) {
	alert("Email address is not valid");

	return false;
  }

  return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
  fieldstocheck[fieldstocheck.length] = value;
  fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
  groupstocheck[groupstocheck.length] = value;
  groupnames[groupnames.length] = name;
}

function compareEmail()
{
  return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}

function checkEmail()
{
  var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.subscribeform.elements["email"].value);
}

function checkGroup(name,value)
{
  option = -1;
  for (i=0;i<document.subscribeform.elements[name].length;i++) {
	if (document.subscribeform.elements[name][i].checked) {
	  option = i;
	}
  }
  if (option == -1) {
	alert ("Please enter your "+value);

	return false;
  }

  return true;
}

</script>
<script type="text/javascript">
function submitForm() {
	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#subscribeform').serialize();
	jQuery.ajax( {
		type: 'POST',
		data: data,
		async: true,
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
		dataType: 'html',
		//contentType: 'multipart/form-data',
		success: function (data, status, request) {
			jQuery("#result").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');
		},
		error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
	});
}

</script>

<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">

	<form method="post" action="" name="subscribeform" id="subscribeform">

		<div class="required"><label for="email">Email address *</label></div>
			<input type=text name=email required="required" placeholder="" size="40" id="email" />

		<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
		<input type="hidden" name="htmlemail" value="1" />

		<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
		<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span>
		<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script>

		<input type="hidden" name="list[2]" value="signup" />
		<input type="hidden" name="listname[2]" value="newsletter"/>
		<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>

	</form>

	<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>

</div>

<script type="text/javascript">

// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';

</script>


</body></html>
user@ose:~$ 
  1. meeting with Marcin about phplist
    1. I explained GDPR and how Alex's site is not properly meeting our obligations
    2. I asked how we should integrate our phplist newsletter to social media, and Marcin said don't worry about it; we should just delete the links to like/retweet the newsletter posting for now. Perhaps in the future we can script a tool to automatically send a campaign when a new post is made to our wordpress site, then post to social media & add-back the "like" and "retweet" links then. But, for now, we just need to get him suited to send newsletters without jumping through 4 distinct gmail accounts.
    3. I asked about attributes. He said that we should import all existing lists' attributes into phplist. therefore, we're going to want to enumerate all of this info into our gdpr-compliant Privacy Policy
    4. I explained how phplist ajax & rest api works (or doesn't work) to add users to our subscription list without requring them to leave out other domain. I'm currently ironing-out final issues in the ajax solution.
    5. I explained how the repermission campaign would be requried after imports.
    6. we discussed how we would have multiple lists one for each of the existing lists, and then we would create one more list for the general "ose newsletter" which only needs the 3 basic & required attributes = timestamp, email address, & accepted PP.


Wed Oct 24, 2018

  1. continuing with my efforts to get a custom form to add a subscriber to our phplist site using AJAX..
  2. the source of offending modsecurity rule id=960010 is in /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
[root@hetzner2 ~]# grep '960010' -B 30 -A 2 /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
# Restrict which content-types we accept.
#
# TODO Most applications support only two types for request bodies
#      because that is all browsers know how to produce. If you are using
#      automated tools to talk to the application you may be using other
#      content types and would want to change the list of supported types.
# 
#      Note though that ModSecurity parses only three content types:
#      application/x-www-form-urlencoded, multipart/form-data request and 
#      text/xml. The protection provided for any other type is inferior.
#
# TODO There are many applications that are not using multipart/form-data
#      types (typically only used for file uploads). This content type
#      can be disabled if not used.  
#
# NOTE We allow any content type to be specified with GET or HEAD
#      because some tools incorrectly supply content type information
#      even when the body is not present. There is a rule further in
#      the file to prevent GET and HEAD requests to have bodies to we're
#      safe in that respect.
#
# NOTE Use of WebDAV requires "text/xml" content type.
#
# NOTE Philippe Bourcier (pbourcier AT citali DOT com) reports
#      applications running on the PocketPC and AvantGo platforms use
#      non-standard content types:
#
#      M-Business iAnywhere      application/x-mal-client-data
#      UltraLite iAnywhere       application/octet-stream
#
SecRule REQUEST_METHOD "!^(?:GET|HEAD|PROPFIND|OPTIONS)$" "phase:1,chain,t:none,block,msg:'Request content type is not allowed by policy',rev:'2',ver:'OWASP_CRS/2.2.9',maturity:'9',accuracy:'9',id:'960010',tag:'OWASP_CRS/POLICY/ENCODING_NOT_ALLOWED',tag:'WASCTC/WASC-20',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/EE2',tag:'PCI/12.1',severity:'2',logdata:'%{matched_var}'" 
		SecRule REQUEST_HEADERS:Content-Type "^([^;\s]+)" "chain,capture"
				SecRule TX:0 "!^%{tx.allowed_request_content_type}$" "t:none,ctl:forceRequestBodyVariable=On,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-OWASP_CRS/POLICY/CONTENT_TYPE_NOT_ALLOWED-%{matched_var_name}=%{matched_var}"
[root@hetzner2 ~]# 
  1. now, I am setting the enctype of the form to "multipart/form-data" in the ajax form, but modsecurity is complaining that it's "text/plain"
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
  1. I found a list of supported data types in /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
[root@hetzner2 ~]# grep 'application/xml' /etc/httpd/modsecurity.d/*
grep: /etc/httpd/modsecurity.d/activated_rules: Is a directory
/etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf:  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json', \
[root@hetzner2 ~]# 
  1. I also found a note that the offending data type (text/plain) was indeed added to the acceptable list in the CRS 3.0 back in 2016; not sure why we don't have that https://github.com/SpiderLabs/owasp-modsecurity-crs/issues/208
  2. so I could update the CRS's config to include "text/plain", but I think the best solution is to fix the AJAX code to send the correct data type.
  3. the offending code is probably somewhere in this jquery.ajax() call (as obtained from the example zip in the phplist forum thread https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
function submitForm() {                                                                                                           
	successMessage = 'Thank you for your registration. Please check your email to confirm.';                                      
	data = jQuery('#subscribeform').serialize();                                                                                  
	jQuery.ajax( {                                                                                                                
		type: 'POST',                                                                                                             
		data: data,                                                                                                               
		url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',                                                     
		dataType: 'html',                                                                                                         
		success: function (data, status, request) {                                                                               
			jQuery("#result").empty().append(data != '' ? data : successMessage);                                                 
			jQuery('#attribute1').val('');                                                                                        
			jQuery('#email').val('');                                                                                             
		},                                                                                                                        
		error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }                
	});                                                                                                                           
}                                                                                                                                 
  1. I've never used jquery, but my guess is that we need to set the "dataType" here; it's documented here https://api.jquery.com/jQuery.ajax/
  2. oh, hmm, actualy reading the docs suggest that the 'dataType' defines what data type we expect _back_ from the server https://api.jquery.com/jQuery.ajax/

The type of data that you're expecting back from the server. If none is specified, jQuery will try to infer it based on the MIME type of the response (an XML MIME type will yield XML, in 1.4 JSON will yield a JavaScript object, in 1.4 script will execute the script, and anything else will be returned as a string). The available types (and the result passed as the first argument to your success callback) are:

  1. so I guess I need to dig into the jquery.serialize() command
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
...
function submitForm() {                                                                                                           
	successMessage = 'Thank you for your registration. Please check your email to confirm.';                                      
	data = jQuery('#subscribeform').serialize(); 
...
}
  1. so the documentation on the serialize function explicitly states that it creates a string https://api.jquery.com/serialize/
  2. ah, so the option I want is "contentType" https://api.jquery.com/jQuery.ajax/
function submitForm() {                                                                               
	successMessage = 'Thank you for your registration. Please check your email to confirm.';          
	data = jQuery('#subscribeform').serialize();                                                          jQuery.ajax( {                                                                                    
		type: 'POST',                                                                                 
		data: data,                                                                                   
		url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',                         
		dataType: 'html',                                                                             
		contentType: 'multipart/form-data',                                                           
		success: function (data, status, request) {                                                   
			jQuery("#result").empty().append(data != '' ? data : successMessage);                     
			jQuery('#attribute1').val('');                                                            
			jQuery('#email').val('');                                                                 
		},                                                                                                    error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }                                                                                          
	});                                                                                               
} 
  1. there's a new modsecurity alert
[Wed Oct 24 20:43:47.876714 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Multipart parsing error (init): Multipart: Boundary not found in C-T. [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]
[Wed Oct 24 20:43:47.876765 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "REQBODY_ERROR" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "151"] [id "960912"] [rev "1"] [msg "Failed to parse request body."] [data "Multipart: Boundary not found in C-T."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]
  1. so this is probably because the content is--in fact--actually "text/plain" there's no multipart boundray since I just lied about what the mime type is. The solution should still be to get serialize() to output the proper mime type
  2. well, it appears that the alternative solution (as is often used for form data including binary content--such as images), is to use the FormData, but this is a browser-specific thing https://developer.mozilla.org/en-US/docs/Web/API/FormData
  3. so proceeding with FormData would likely limit the browsers in which this ajax form will work. It's really not requred for the small fields needed for subscriber attributes data in this form. so let's just ignore the modsecurity role
  4. well, I tried to overwrite the existing allowable mime types in our vhost file (making restore from backups easier). that didn't work
  5. I tried to just add the rule id = 960010 to the long list of SecRuleRemoveById list in the vhost file; that also didn't work
  6. the only way it worked as if I actually edited the list in the /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf file, adding "text/plain" to the list (followed by an apache reload) https://stackoverflow.com/questions/41604674/customizing-apache-mod-security-to-accept-content-type-text-plain
[root@hetzner2 modsecurity.d]# pwd
/etc/httpd/modsecurity.d
[root@hetzner2 modsecurity.d]# grep -C5 'text/plain' modsecurity_crs_10_config.conf
SecAction \
  "id:'900012', \
  phase:1, \
  t:none, \
  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS', \
  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json|text/plain', \
  setvar:'tx.allowed_http_versions=HTTP/0.9 HTTP/1.0 HTTP/1.1', \
  setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/', \
  setvar:'tx.restricted_headers=/Proxy-Connection/ /Lock-Token/ /Content-Range/ /Translate/ /via/ /if/', \
  nolog, \
  pass"
[root@hetzner2 modsecurity.d]# 
  1. so this time it just got a 200 from the server (logs all look clean), but the user didn't get added. The js console shows some (unrelated?) errors about our jquery script file
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More]
jquery.min.js:112:77
SyntaxError: missing ) after argument list[Learn More]
phplistAjax.php:1:58
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
phplistAjax.php:7:2
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2
ready
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
  1. I checked out the sourcecode of our phplist subscribe page, and I discovered that it has its own checkform() js function, and some helper functions. I just copied these to my ajax form https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
  2. unrealted, but I saw that at the top of this sourcecode is a php command that for some reason came out as text, not processed by php
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phpList version 3.3.3" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" href="./images/phplist.ico" />
<title>Subscribe to our Newsletters</title><meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<meta name="theme-color" content="#2C2C2C"/>
<link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" />
<link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" />
<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />
</head>
  1. it looks like this comes from pagetop_minified.php
[root@hetzner2 lists]# pwd
/var/www/html/phplist.opensourceecology.org/public_html/lists
[root@hetzner2 lists]# cat admin/ui/dressprow/pagetop_minified.php 
<?php
/*
  We request you retain the full headers below including the links.
  This not only gives respect to the large amount of time given freely
  by the developers, but also helps build interest, traffic and use of
  phpList, which is beneficial to it's future development.

  Michiel Dethmers, phpList Ltd 2003 - 2015
*/
?>
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="<?php echo $_SESSION['adminlanguage']['iso']?>" lang="<?php echo $_SESSION['adminlanguage']['iso']?>" dir="<?php echo $_SESSION['adminlanguage']['dir']?>">
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phplist version <?php echo VERSION?>" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" id="favicon" href="./images/phplist.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<link rel="stylesheet" href="ui/dressprow/css/all.min.css?v=<?php echo filemtime(dirname(FILE).'/css/all.min.css'); ?>" />

<?php
if (isset($GLOBALS['config']['head'])) {
	foreach ($GLOBALS['config']['head'] as $sHtml) {
		echo $sHtml;
		echo "\n";
		echo "\n";
	}
}
[root@hetzner2 lists]# 
  1. interstingly, there's php lines above it (ie: the "phplist version") execute without issues
  2. this is somewhat related, but not really https://mantis.phplist.org/view.php?id=18447
  3. ah, this is it https://mantis.phplist.org/view.php?id=19307
    1. looks like the fix was to just remove it https://github.com/phpList/phplist-ui-bootlist/pull/62
  4. anyway, it looks like the error "SyntaxError: missing ) after argument list" was caused by my attribute with a quote in it. Xheni warned me about this here https://mantis.phplist.org/view.php?id=19436
  5. I replaced my links' encapsulating double qoutes with single quotes as a temp fix
  6. ok, better fix, update the attribute to use single quotes in phplist's admin ui
    1. so actually the issue is quotes, not slashes.
  7. ok, that elminated the error entirely. Now I just have
Loading mixed (insecure) display content “http://powered.phplist.com/images/3.3.3/power-phplist.png” on a secure page[Learn More]
jquery.min.js:112:77
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
  1. this is frustrating, but I confirmed that signup *does* work from our actual subscribe page https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1
  2. I noticed that there were a few js files included in the phplist site's code, so I added these to my ajax form
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>                                                                 
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>                                                                                               
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>       
  1. now I get a pop-up alert = "Sorry, we were unable to process your subscription."
  2. someone else posted about this issue a couple years ago https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
  3. ah, another point. I have the URL wrong! per https://discuss.phplist.org/t/ajax-subscribe-api/974/15
    1. I had been using https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1
url: 'https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1',
    1. but, in fact, it should use 'asubscribe' intead of 'subscribe' = https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=1
  1. per the thread above, I also changed it to default to html & not confirm email address
  2. it's still not working. this is taking for fucking ever. I gotta get some POC. I'll simplify it by stripping out the "I agree" checkbox for now. If I can fucking get that to work, I'll try to add a simple checkbox and work from there..
  3. I created a new list and cooresponding subscribe page that's as simple as possible: it's just an email address https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=2
  4. ugh, even that won't work.
  5. oh, I got it to work! I had to set both the name and the id attributes of the form field to "subscribeform". Note that the form coming from phplist's html only sets the 'name' attribute, but the jquery serialize() function is called against the 'id' attribute. otherwise, it probably just sent an empty string to the subscribe page!
<form method="post" action="" name="subscribeform" id="subscribeform">
  1. woohoo! I switched back to the list=1 (with the checkbox agreeing to our PP), and it worked! When the checkbox is *not* ticked, a JS pop-up yells at you:

The following field is required: I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.

    1. unfortunately, it still says that it fails even when it succeeds, as the thread above pointed out here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
  1. fwiw, this is what woked today. Tomorrow, I should look into making this visible only if JS is enabled. Otherwise, I should just display an anchor link the user to our phplist subscription page
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html                              100% 5329    53.4KB/s   00:00    
user@ose:~$ cat phplistAjax.html 
<html><body>

<head>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<script type="text/javascript">

function checkform()
{
  for (i=0;i<fieldstocheck.length;i++) {
	if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
	  if (document.subscribeform.elements[fieldstocheck[i]].checked) {
	  } else {
		alert("The following field is required:  "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	} else {
	  if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
		alert("Please enter your "+fieldnames[i]);
		eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");

		return false;
	  }
	}
  }
  for (i=0;i<groupstocheck.length;i++) {
	if (!checkGroup(groupstocheck[i],groupnames[i])) {
	  return false;
	}
  }
  
  if (! checkEmail()) {
	alert("Email address is not valid");

	return false;
  }

  return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
  fieldstocheck[fieldstocheck.length] = value;
  fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
  groupstocheck[groupstocheck.length] = value;
  groupnames[groupnames.length] = name;
}

function compareEmail()
{
  return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}

function checkEmail()
{
  var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.subscribeform.elements["email"].value);
}

function checkGroup(name,value)
{
  option = -1;
  for (i=0;i<document.subscribeform.elements[name].length;i++) {
	if (document.subscribeform.elements[name][i].checked) {
	  option = i;
	}
  }
  if (option == -1) {
	alert ("Please enter your "+value);

	return false;
  }

  return true;
}

</script>
<script type="text/javascript">
function submitForm() {
	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#subscribeform').serialize();
	jQuery.ajax( {
		type: 'POST',
		data: data,
		async: true,
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
		dataType: 'html',
		//contentType: 'multipart/form-data',
		success: function (data, status, request) {
			jQuery("#result").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');
		},
		error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
	});
}

</script>
</head>

<form method="post" action="" name="subscribeform" id="subscribeform"><table border=0>
  <tr><td><div class="required"><label for="email">Email address *</label></div></td>
  <td class="attributeinput"><input type=text name=email required="required" placeholder="" size="40" id="email" />
  <script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script></td></tr><input type="hidden" name="htmlemail" value="1" />
<tr><td colspan="2">
<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span><script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script></td></tr>
</table><input type="hidden" name="list[2]" value="signup" /><input type="hidden" name="listname[2]" value="newsletter"/><div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div><input type=submit name="subscribe" value="Subscribe to the selected newsletters" onClick="return checkform();">    <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website" ><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0" /></a></p></div>

<button class='button' 
	onclick="if (checkform()) {submitForm();} return false;"
>Subscribe</button>
<div id="result" style="color: red;"></div>

</div>



</body></html>
user@ose:~$ 

Tue Oct 23, 2018

  1. Sam got back to me about integrating a signup list into wordpress via AJAX or the REST API. They said the API doesn't yet support attributes (that conflicts with the example I saw yesterday, though). Sam said adding that feature to the API will come with time, but we could fund it to be built in 6-8 weeks for ~$4,500. Of course, we could also just develop it ourselves and PR it into their repo (it uses Symfony and Doctrine) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/4
    1. Sam didn't give a direct answer, but suggested that it's probably possible to do with ajax
  1. fuck it; let's see if I can manually create a form that does what I want with ajax. I'll use this old example as a template https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
  2. I put the file up on the microfactory site in the wp-content dir.
  3. I got a success message (Thank you for your registration. Please check your email to confirm.), but it didn't add to the subscribers list. I checked the html console, and I got an error due to Cross-Origin
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/?p=subscribe&id=1. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘http://phplist.opensourceecology.org’).
  1. I found that phplist inclues logic to define the "Acces-Control-Allow-Origin" header based on a config file option
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'Allow-Origin' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'ACCESS_CONTROL_ALLOW_ORIGIN' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
public_html/lists/admin/init.php:if (!defined('ACCESS_CONTROL_ALLOW_ORIGIN')) {
public_html/lists/admin/init.php:    define('ACCESS_CONTROL_ALLOW_ORIGIN', $GLOBALS['scheme'].'://'.$_SERVER['HTTP_HOST']);
[root@hetzner2 phplist.opensourceecology.org]# 
  1. and here's the relevant option in the config file https://resources.phplist.com/system/config/access_control_allow_origin
  2. apparently the protcol (https://) must be specified. I finally got a new error after adding this line to the phplist config.php file
// allow AJAX queries to add subscribers to our db from other domains                                                      
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );
  1. now the error is
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More]
jquery.min.js:112:77
SyntaxError: missing ) after argument list[Learn More]
phplistAjax.php:1:58
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/
jquery.min.js:127:343
TypeError: $(...).tagsinput is not a function[Learn More]
phplistAjax.php:7:2
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2
ready
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208
<anonymous>
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1
globalEval
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285
Qa
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201
each
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149
domManip
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185
append
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419
success
https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13
b
https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1
ajax/x.onreadystatechange
  1. this time jquery just fucking fetched the whole form and put it on the page somehow.
<div class="panel">
<br>
<div class="content  well"><h3>Subscribe to our newsletters</h3>

<div class="error"><span class="required">* required fields are marked red</span></div>
<div class="error missing">The following required values are missing: Email addresses you entered do not match</div><br>

<form method="post" action="" name="subscribeform"><table border="0">
  <tbody><tr><td><div class="required"><label for="email">Email address *</label></div></td>
  <td class="attributeinput"><input name="email" required="required" placeholder="testa@example.com" size="40" id="email" type="text">
  </td></tr>
  <tr><td><div class="required"><label for="confirm">Confirm your email address *</label></div></td>
  <td class="attributeinput"><input name="emailconfirm" required="required" value="" size="40" id="confirm" type="text">
  </td></tr><tr><td colspan="2">
	  <span class="attributeinput">
	  <input name="textemail" value="1" id="textemail" type="checkbox"></span>
	  <span class="attributename"><label for="textemail">I prefer to receive emails in Text format</label></span>
	  </td></tr>
<tr><td colspan="2">
<input name="attribute4" value="on" class="attributeinput" id="attribute4" type="checkbox">
<span class="required"><label for="attribute4">I agree to the OSE <a href="https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy">Privacy Policy</a>. *</label></span></td></tr>
</tbody></table><input name="list[2]" value="signup" type="hidden"><input name="listname[2]" value="newsletter" type="hidden"><div style="display:none"><input name="VerificationCodeX" value="" size="20" type="text"></div><input name="subscribe" value="Subscribe to the selected newsletters" onclick="return checkform();" type="submit">    <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website"><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0"></a></p></div>
</div>
  1. I used this html form data to merge into the example one
  2. I hit a modsecurity issue with rule #960010
Request content type is not allowed by policy
    1. I think it may be related to the content type submitted by the form somewhere defaulting to "text/plain" *shrug*
  1. I had lost my password for our phplist install due to my ssd crash; I reset it (noting that you cannot login with your email address; you must use your username--as addressed in the password reset email. In my case, 'maltfield').
  2. I also added an admin for marcin
  3. I confirmed that the shared 'admin' user is in our shared ose keepass db

Mon Oct 22, 2018

  1. Marcin sent me our other 3 email spreadsheets; I dug through them to see what attributes we collected
  2. according to phplist plugin page, the rest api doesn't support setting attributes https://resources.phplist.com/plugin/restapi
    1. however, there does appear to be support for non-email attributes per this document, even though it does mention on the main page of this github repo that it's deprecated (the replacement for php4 isn't stable, however) https://github.com/phpList/phplist-plugin-restapi/blob/master/plugins/restapi2/docs/api.apib#L185-189
  3. the replacement api's repo does say that it can work for php3 https://github.com/phpList/rest-api

This new REST API can also be used to provide REST access to an existing phpList 3 installation. For this, the phpList 3 installation and the phpList 4 installation with the REST API need to share the same database. For security reasons, the REST APIs from phpList 3 and phpList 4 should not be used for the same database in parallel, though.

Sat Oct 20, 2018

  1. I did a search for wordpress plugins using the keyword "phplist". Here's some noteable results
    1. Sign Me Up https://wordpress.org/plugins/sign-me-up/
      1. - last updated 3 years ago
      2. + explicitly lists that it connects with phplist using AJAX
      3. - says that it doesn't support additional attributes (email address only). This will probably not work as we need to at least have a checkbox for the user to accept our gdpr-ready Privacy Policy
      4. there's a lot of info about this plugin here https://www.jaromy.net/wordpress-plugins/sign-me-up[[1]]
    2. WP PHPList https://wordpress.org/plugins/phplist-form-integration/
      1. - last updated 9 years ago
    3. yPHPlista https://wordpress.org/plugins/yphplista/
      1. - last updated 8 years ago https://wordpress.org/plugins/yphplista/
  1. I tried searching the phplist forums for "wordpress" to dig through recent discussions about integrating phplist with wordpress
    1. I found this interesting discussion hinting to having a phplist plugin automatically send email campaigns after a new wordpress post is in. I really like this idea (ie: RSS scraping -> email campaign). Prior to sending the email campaign, we could post a link to it on facebook, twitter, etc. Then we could add a link to "like" or "retweet" the post (on social media) to the original post (on wordpress) within the email template. https://discuss.phplist.org/t/how-is-content-generated-in-wordpress/1656/6
    2. so 6 months ago someone asked my question: is there an official wp plugin for phplist? Sam Tuke (CEO of phplist) responded stating that there isn't but it's needed. And expounded on how the AJAX solution is the old way and that the future phplist wp plugin would work via the REST API (I like this much better too) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894
    3. this may work with the AJAX way https://discuss.phplist.org/t/wordpress-integration-newsletter-sign-up-plugin/1723/15
    4. And there was this old post for 2016 about the 9-year-old "WP PHPList" (phplist-form-integration) wordpress plugin https://discuss.phplist.org/t/phplist-wordpress-integration-solved/2279/2
  1. I lost my password for phplist's discourse site due to my ssd crash, so I generated a new one
  2. Interestingly, it looks like phplist's discourse site is SSO using wordpress as the account backend
  3. I replied to this post, asking Sam if there's an option for sending our new subscriber's attributes to the stable version phplist via either AJAX or REST https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/3

Fri Oct 19, 2018

  1. I'm back in NYC after restoring my laptop following my ssd crash
  2. sent Sara information about wp-cli and managing wordpress
    1. I also sent an email to Marcin asking if he authorizes them having ssh and/or shraed ose keepass access
    2. specifically, the plugin that Sara wanted to install was this https://wordpress.org/plugins/custom-css-js
    3. the above plugin is freemium. the reviews suggested this free alterantive for more features, but it's a less popular plugin https://wordpress.org/plugins/scripts-n-styles/
  3. Marcin sent me a confusing email about utilization of google analytics. One of our first conversations with Marcin back in 2017 was about how OSE was working to migrate _away_ from Google services. We've since stripped analytics from all our sites. We looked at Piwik, but--due to security issues--we decided to go with awstats.
    1. I restated that we use awstats, and asked what are the benefits of using google analytics
    2. I added the microfactory site to our awstast generation cron job, and confirmed that I could view all the stats so far this month https://awstats.opensourceecology.org:4443/awstats.microfactory.opensourceecology.org.html
    3. In general, I'm very confused about where the line is drawn between giving up on FLOSS and moving to some closed-source SaaS solution here at OSE.
      1. I've been hacking the hell out of phplist so that we can use an ugly FLOSS solution, but Mailchimp would be much better, and very cheap for our needs.
      2. We still don't have a decent issue tracker. Mantis is a long-standing TODO, but we could use Jira for free because we're a nonprofit.
      3. Google Analytics is surely more powerful thatn awstats, but does that really justify the use of closed-source tools that exploit our visitor's data? Why draw the line here between awstats & GA but not use Jira or Mailchimp like most other nonprofits?
  4. Marcin sent me an error when trying to post our email thread about installing a wordpress plugin to the wiki
    1. fixed by modsec whitelist
      1. 950019, generic attack = email injection
  5. I updated the documentation on the wiki for the quick-start guide to configuring 2FA for our wordpress sites using the Google Authenticator plugin https://wiki.opensourceecology.org/wiki/2FA#Quick_Start_Guide
    1. I added a step to enable relaxed mode, so the user's phone only has to be correct within 4 minutes, rather than 30 seconds.
    2. I recommended use of the oandbackup app, which is the FLOSS replacement for Titanium Backup https://f-droid.org/en/packages/dk.jens.backup/
  1. ...
  1. began to revisit my install of the "Newsletter Sign-Up" wordpress plugin on the fef site
  2. I logged into the fef wordpress ite
  3. I navigated to the Plugins -> "Newsletter Sign-Up" -> "Settings" link (note that this redirects to the main panel nav bar's "Newsl. Sign-Up" link.
    1. I changed "Select your mailinglist provider" to "PHPList"
    2. our subscriber link is this, so I left the "PHPList list ID" at "1" https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
    3. I set the "Newsletter form action" to "https://phplist.opensourceecology.org/lists/?p=subscribe&id=1"
  4. this plugin appears to be mostly built around the trick of having someone signup when leaving a comment. That's a feature that we may want to disable. Mostly we want a widget and a form embedded into a post/page.
    1. I picked a bad site, fef, for testing widgets. it doesn't use them!
    2. I tested adding a form for signing up to the phplist newsletter to the lola page by adding "[nsu_form]" to the content of the post https://fef.opensourceecology.org/2015/09/03/lola/
      1. when I submitted on this form, it just fucking redirected me to the phplist site. it didn't even auto-fill the form! what rubbish! what we want here is for a seamless submittion to the phplist site without the user having to leave the site that they are on.
  5. ugh, I just realized that the "Newsletter Sign-Up" form's main plugin page on the wp site shows a message indicating that the plugin is no longer maintained (it was last updated 11 months ago) https://wordpress.org/plugins/newsletter-sign-up/
  6. I deactivated the "Newsletter Sign-Up" plugin and deleted the related content & widgets from the fef site.
  7. looks like I lost my logs from when I discovered the "Newsletter Sign-Up" and similar wordpress plugins
  8. I dug up some options
    1. https://wordpress.org/plugins/sign-me-up/
      1. - last updated 3 years ago
    2. https://wordpress.org/plugins/yphplista/
      1. - last updated 8 years ago
  9. it looks like phplist.com (not .org) offers a solution to this issue, the so-called "AJAX" solution so subscribres can signup "...and stay on your website" https://www.phplist.com/ajaxdemo
    1. found some info on enabling this the '.org' way. note that there may be issues with the ACCESS_CONTROL_ALLOW_ORIGIN https://discuss.phplist.org/t/ajax-subscribe-api/974/7
define('ACCESS_CONTROL_ALLOW_ORIGIN', 'www.mydomain.de');
define('PUBLIC_PROTOCOL','https');
  1. I found a few resources about ajax on phplist.com's wiki https://resources.phplist.com/start?do=search&id=ajax
    1. ugh, this just references the 2x no-longer-maintained plugins I found above https://resources.phplist.com/develop/wordpress?s[]=ajax
  2. there's only 1x reference to "ajax" in the phplist.org manual it referes to the Ethical Pet case study. Yeah, the one that migrated off from phplist..
  3. important, but a fucking stub. can we get an example listing 3x domains? https://resources.phplist.com/system/config/access_control_allow_origin?s[]=ajax
  4. the discuss thread above includes this example file from subscription_form_example.html (in subscription_form_example.html.zip) https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
    1. ugh, the first line includes the jquery src @ googleapis.com. Let's not make our users require decentraleyes to prevent google from tracking them on our site..
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
  1. it's looking like the best option is to use the no longer maintained "Newsletter Sign-Up" wp plugin still :\ I re-enabled it and added the line "[nsu_form]" to the lola paga again. I'll try to debug that before reinventing the wheel https://fef.opensourceecology.org/2015/09/03/lola/
    1. I checked the code; there's not even any JS; it's just a fucking form with submit action to our distinct phplist site (I figured it was some ajax cross-site rejection failure causing fall-back to the redirect). Maybe this plugin isn't what we want still..I disabled it again.


Sun Oct 14, 2018

  1. bought & installed my replacement ssd after my old one died unexpectedly
  2. install os, restore backups, etc
  3. catching up on old email

Mon Oct 01, 2018

  1. Note that my logs for the first few weeks of this quarter were lost when my laptop's ssd crashed. Please mind the gap.