Maltfield Log/2018 Q4
Jump to navigation
Jump to search
My work log from the year 2018 Quarter 4. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
See Also
Sun Nov 07, 2018
- Duncan Cameron noted that in their environment, my change to throw an exception didn't change the exit code to 200. I looked it up, and found that if display_errors is enabled, then it will stay at 200--otherwise it's 500. https://github.com/phpList/phplist3/pull/432
- I replied to this thread https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-subscribelib2-php/4578
Sun Nov 04, 2018
- the phplist team approved & merged my PR to throw an exception when the ajax subscribe fails on the server, which changes the https status code from 200 ok to 500 internal server error https://github.com/phpList/phplist3/pull/432
- continuing with why the subscribelib2.php script thinks our ajax subscribe is a "hack attempt" and exits prematurely..
- it looks like it's because the the key in the key/value pair from the "list[2]=subscribe" argument is empty.
error_log( "made it to AC" );
if (isset($_POST['list']) && is_array($_POST['list'])) {
foreach ($_POST['list'] as $key => $val) {
error_log( "\tlist key|$key| val|$val|" ); if ($val == 'signup') {
$key = sprintf('%d', $key);
if (!empty($key)) {
error_log( "made it to AAA" );
$result = Sql_query(sprintf('replace into %s (userid,listid,entered) values(%d,%d,now())',
$GLOBALS['tables']['listuser'], $userid, $key));
$lists .= "\n * ".listname($key);
$subscriptions[] = $key;
addSubscriberStatistics('subscribe', 1, $key);
} else {
//# hack attempt...
error_log( "made it to AAB" );
exit;
}
}
}
}
error_log( "made it to AD" );
- the above code yields the following in the error log, indicating that the loop itreates through two distinct elements in the $_POST['list'] array. One's key is '2' and the other is empty.
[Sun Nov 04 18:56:57.061936 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AC, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.061951 2018] [:error] [pid 14919] [client 127.0.0.1:44604] \tlist key|2| val|signup|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.061977 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AAA, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.069666 2018] [:error] [pid 14919] [client 127.0.0.1:44604] \tlist key|| val|signup|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php [Sun Nov 04 18:56:57.069708 2018] [:error] [pid 14919] [client 127.0.0.1:44604] made it to AAB, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
- I created a thread on the phplist discourse forms asking why that "hack attempt" exit exists, and describing how it's breaking our ajax subscription flow via jquery.ajax(). I asked if I can remove it in a PR if it's not necessary https://discuss.phplist.org/t/purpose-of-hack-attempt-exit-in-libsubscribe2-php/4578
- upon further research, I found that my ajax form is indeed submitting only "list[2]=signup" and that the "list[]=signup" paramater is actually being added by phplist (!) here https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/index.php#L247-L249
- I'm not sure what the $GLOBALS['pagedata']['lists'] data *should* have. In any case, I updated the discourse thread linked above
- in the meantime, I commented-out the "hack attempt" comment and encapsulating else statement
- now the subscribelib2.php code is ending at the "return 2" line here https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/admin/subscribelib2.php#L407
- ok, so now when I send the right inputs, the result div gets populated with the "thanks, you have been added to the newsletter..." message as returned by the phplist server's response. Horray! I also confirmed that when I submit with bad data, it fails properly (with a pop-up saying "Sorry, we were unable to process your subscription."
- it doesn't actually clear the success message after the fail message comes back, so that needs to be fixed next week
- ...
- I did some further digging into the code that checks for required attributes, regarding this bug I submitted earlier this week https://mantis.phplist.org/view.php?id=19503
- it appears that the first part of the if statement checks $subscribeagedata rather than checking the DB. I'm not sure what the advantage of this is, except for maybe making some attributes optional on some subscribe pages. The issue in my case was that my attribute in the relevant subscribe page (as determined by the GET variable (id=2) had the ToS attribute's "Is this attribute required" checkbox ticked, but then the "Check this box to use this attribute in the page" checkbox was *not* ticked. https://phplist.opensourceecology.org:4443/lists/admin/?page=spageedit&tk=2d2c23511c2b9762c09026d77c25cd31&id=2
- when I _did_ check the "use this attribute" checkbox (in addition to the "required" checkbox, which was already checked), the $subscribepagedata finally included the ToS attribute4 in the list of required attributes
[Sun Nov 04 19:41:22.044846 2018] [:error] [pid 30671] [client 127.0.0.1:58448] subscribepagedata:|Array\n(\n [ajax_subscribeconfirmation] => <h3>Thanks, you have been added to our newsletter</h3><p>You will receive an email to confirm your subscription. Please click the link in the email to confirm</p>\n [attribute004] => 4######0###1\n [attributes] => 4+\n [button] => Subscribe to the selected newsletters\n [emaildoubleentry] => no\n [footer] => </div>\r\n</div>\r\n</div><!-- ENDOF #mainContent-->\r\n</div><!-- ENDOF .wrapper -->\r\n</div><!-- ENDOF #container -->\r\n\r\n<div id="footer">\r\n<div id="footerframe">\r\n<ul class="list-unstyled">\r\n<li> </li>\r\n</ul>\r\n</div>\r\n</div>\r\n<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>\r\n<script type="text/javascript" src="admin/js/phplistapp.js"></script>\r\n<script type="text/javascript" src="admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>\r\n<script>\r\n/* do not remove this from here */\r\n$(document).ready(function(){\r\n if ( $('body').hasClass('invisible') ){\r\n myfunction();\r\n }\r\n\t$('#edit_list_categories input.form-control').tagsinput('refresh');\r\n\r\n});\r\n</script>\r\n</body>\r\n\r\n</html>\r\n\n [header] => <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />\r\n<meta name="theme-color" content="#2C2C2C"/>\r\n<link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" />\r\n<link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" />\r\n<link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" />\r\n</head>\r\n\r\n<body class="fixed invisible">\r\n\r\n<div id="container">\r\n\r\n<div id="header" class="navbar navbar-inverse">\r\n<div id="rack-functions" class="container">\r\n\t<h1 id="logo"><a href="http://phplist.opensourceecology.org" title="Visit our website">Open Source Ecology</a></h1>\r\n</div>\r\n</div>\r\n<div id="wrapper" class="container">\r\n<div id="mainContent">\r\n\r\n<div class="panel">\r\n<br />\r\n<div class="content well">\n [htmlchoice] => htmlonly\n [intro] => <h3>Subscribe to our newsletters</h3>\n [language_file] => \n [thankyoupage] => <h3>Thank you for subscribing to our newsletters.</h3>\r\nYour email address has been added to our system. You will receive a message with a request to confirm your membership. Please make sure to click the link in that message to confirm your subscription.\n [title] => poc\n [lists] => \n [rssdefault] => \n [rssintro] => \n [rss] => \n)\n|, referer: https://microfactory.opensourceecology.org/wp-content/phplistAjax.php
- I updated the bug report with my UX change suggestion
Fri Nov 02, 2018
- when I was poking at the ajax form & server-side code, I found that the ajax's "error" function was successfully called when I had a php-side syntax error, causing a 500 (internal server error) to be returned. So that's proven to work. It still does not, however, catch the FAIL response, which is included in a 200 response. I supose I could either fix the php code to return a non-200 (and submit a PR) or just catch FAIL messages from the ajax success() function and then call the ajax error() function from within success().
- I decided that the best way to handle this is for the php script to throw an exception instead of just print "FAIL", which causes a 500 status to be returned & is caught by the AJAX code
- I created a bug report for this here https://mantis.phplist.org/view.php?id=19524
- ugh, I discovered that phplist created their own sqllog() function in mysqli.inc, and then hard-codes log files to be written to "/tmp/". I wrote a post to the phplist Discourse forum askint the developers why they avoid php's built-in error_log() function https://discuss.phplist.org/t/why-doesnt-phplist-use-error-log/4570
- Xheni responded stating that they'd rather not spam the error log, and instead create a new log file in a new dir and lock it down with php. I stated that I don't think that's really possible, and that using trigger_error() would allow you to set messages as E_STRICT--which actually would not be written to the log file by default (only done so if the logging verbosity was increased on, say, a dev server). I also mentioned that they should use the php function sys_get_temp_dir() to get php's upload_tmp_dir rather than expect to be able to access '/tmp' as it's hardcoded now--since open_basedir may deny access to '/tmp/'.
- I reiterated that using trigger_error() would be able to send the messages to a log file that was readable only to root, something that I don't think would be possible to achieve within a php script.
- I finally found out what the form input VerificationCodeX is for. Apparently we just create it as a hidden field and do nothing with it. I gues the hope is that spam bot will attempt to fill it in. If it's anything but empty, we reject it as spam. I do something similar with an input field on email.michaelaltfield.net https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/admin/subscribelib2.php#L92-L119
- for some reason when subscfribelib2.php encounters an error, it returns lots of info, including the sucess messages. But when it successfully adds my user, it returns a generic payload without the success message.
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" > <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr"> <head> <meta http-equiv="pragma" content="no-cache" /> <meta http-equiv="Cache-Control" content="no-cache, must-revalidate" /> <meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" /> <meta name="Author" content="Michiel Dethmers - http://www.phplist.com" /> <meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" /> <meta name="Powered-By" content="phpList version 3.3.3" /> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <link rel="SHORTCUT ICON" href="./images/phplist.ico" />
- it appears that subscribelib2.php ends early (exit? return?). I should find out why. The goal here is to get phplist to return a 200 with the phplist configured success message on success. And it should return a 500 on failure, hopefully with some message as to why (though I'd accept a missing explination at this point)
- looks like I'm being detected as a hacker? I found that It's exiting just after the line "hack attempt" further research is required. https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/admin/subscribelib2.php#L255-L256
if (isset($_POST['list']) && is_array($_POST['list'])) {
foreach ($_POST['list'] as $key => $val) {
if ($val == 'signup') {
$key = sprintf('%d', $key);
if (!empty($key)) {
error_log( "made it to AAA" );
$result = Sql_query(sprintf('replace into %s (userid,listid,entered) values(%d,%d,now())',
$GLOBALS['tables']['listuser'], $userid, $key));
$lists .= "\n * ".listname($key);
$subscriptions[] = $key;
addSubscriberStatistics('subscribe', 1, $key);
} else {
//# hack attempt...
error_log( "made it to AAB" );
exit;
}
}
}
} error_log( "made it to AD" );
Tue Oct 30, 2018
- continuing with my quest to get the ajax form to work properly..
- I thought maybe the root cause of the empty error was the js console's message about CORS issues
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at ‘https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2’. (Reason: Credential is not supported if the CORS header ‘Access-Control-Allow-Origin’ is ‘*’).
</b>
# I changed the phplist config file from allowing '*' to just our domain
<pre>
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );
- but then the error changed to one about the Credentials issue
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2. (Reason: expected ‘true’ in CORS header ‘Access-Control-Allow-Credentials’).
- I did some research on this specific error, and it appears to pop-up when both the server and the client don't explicitly permit the Credentials flag, which would allow the server to set cookies across domains, creating possible CSRF vulnerabilities https://stackoverflow.com/questions/24687313/what-exactly-does-the-access-control-allow-credentials-header-do
- I checked the debugger for cookies being stored, and--sure enough--there was one named 'browsetrail'. I searched the entire phplist vhost dir for this file, and found a few releavant files. Namely, I found one of the js files we were including at the top of our script = phplistapp.js
[root@hetzner2 phplist.opensourceecology.org]# grep -irl 'browsetrail' * public_html/lists/admin/connect.php public_html/lists/admin/ui/dressprow/js/all.js public_html/lists/admin/ui/dressprow/js/all.min.js public_html/lists/admin/ui/default/js/all.js public_html/lists/admin/ui/default/js/all.min.js public_html/lists/admin/js/phplistapp.js [root@hetzner2 phplist.opensourceecology.org]#
- so I commented-out the include of this in our script, and the CORS erros went away entirely
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<!-- trying to prevent the 'browsetrail' cookie causing CORS errors in Access-Control-Allow-Credentials
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
-->
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
- After some digging, I found strikingly little logic behind the ajax backend that's listening to our form's requests. It's just this https://github.com/phpList/phplist3/blob/7ec8ab78f215894c29da8e54ff9d7c41ffde64c6/public_html/lists/index.php#L242-L272
case 'asubscribe': //# subscribe with Ajax
$_POST['subscribe'] = 1;
if (isset($_GET['email']) && !isset($_POST['email'])) {
$_POST['email'] = $_GET['email'];
}
foreach (explode(',', $GLOBALS['pagedata']['lists']) as $listid) {
$_POST['list'][$listid] = 'signup';
}
$_POST['htmlemail'] = 1; //# @@ should actually be taken from the subscribe page data
$success = require 'admin/subscribelib2.php';
$result = ob_get_contents();
ob_end_clean();
if (stripos($result, $GLOBALS['strEmailConfirmation']) !== false ||
stripos($result, $pagedata['thankyoupage']) !== false
) {
if (!empty($pagedata['ajax_subscribeconfirmation'])) {
$confirmation = $pagedata['ajax_subscribeconfirmation'];
} else {
$confirmation = getConfig('ajax_subscribeconfirmation');
}
if (empty($confirmation)) {
echo 'OK';
} else {
echo $confirmation;
}
exit;
} else {
echo 'FAIL';
}
break;
- so basically all I have to work with is if it responded with 'FAIL' or 'OK'. In both cases, the jquery ajax function will hook the "success" function.
- this is very bad as we're basically depending on our js form for proving that the user accepted the Privacy Policy. Indeed, if I remove the JS on the form that ensures the "I agree..." checkbox is ticked and submit the form with it unticked, then phplist happily subscribes the user! This is *wrong*!
- the best solution, then, is to update this code
- unfortunaetly, the ajax solution that we're updating is being deprecated in the next release of phplist4 in favor of the REST API
- the attribute that I need to verify identifies itself as "attribute4" — which is hardly specific
- note, however, that the attribute does have the property (meta attribute?) "is this attribute required"
- therefore, probably the best way to write this robustly is to get a list of all the attributes that are requried & ensure that they are present. If any required attributes are not present (or have 0 /blank/ null values), then do *not* subscribe the user and return an error
- so, actually, the block above does require another script--which actually does the logic for adding the user. Indeed, there is some logic in there for checking for required fields. https://github.com/phpList/phplist3/blob/19baaaa476bb532f53430ff9b3e7fd73e61991a1/public_html/lists/admin/subscribelib2.php#L18=L81
//# Check if input is complete
$allthere = 1;
$subscribepagedata = PageData($id);
if (isset($subscribepagedata['language_file']) && is_file(dirname(FILE).'/../texts/'.basename($subscribepagedata['language_file']))) {
@include_once dirname(FILE).'/../texts/'.basename($subscribepagedata['language_file']);
}
// Allow customisation per installation
if (is_file($_SERVER['DOCUMENT_ROOT'].'/'.basename($GLOBALS['language_module']))) {
include_once $_SERVER['DOCUMENT_ROOT'].'/'.basename($GLOBALS['language_module']);
}
if (!empty($data['language_file']) && is_file($_SERVER['DOCUMENT_ROOT'].'/'.basename($data['language_file']))) {
include_once $_SERVER['DOCUMENT_ROOT'].'/'.basename($data['language_file']);
}
$required = array(); // id's of missing attribbutes '
if (count($subscribepagedata)) {
$attributes = explode('+', $subscribepagedata['attributes']);
foreach ($attributes as $attribute) {
if (isset($subscribepagedata[sprintf('attribute%03d',
$attribute)]) && $subscribepagedata[sprintf('attribute%03d', $attribute)]
) {
list($dummy, $dummy2, $dummy3, $req) = explode('###',
$subscribepagedata[sprintf('attribute%03d', $attribute)]);
if ($req) {
array_push($required, $attribute);
}
}
}
} else {
$req = Sql_Query(sprintf('select * from %s', $GLOBALS['tables']['attribute']));
while ($row = Sql_Fetch_Array($req)) {
if ($row['required']) {
array_push($required, $row['id']);
}
}
}
if (count($required)) {
$required_ids = implode(',', $required);
// check if all required attributes have been entered;
if ($required_ids) {
$res = Sql_Query("select * from {$GLOBALS['tables']['attribute']} where id in ($required_ids)");
$allthere = 1;
$missing = '';
while ($row = Sql_Fetch_Array($res)) {
$fieldname = 'attribute'.$row['id'];
$thisonemissing = 0;
if ($row['type'] != 'hidden') {
$thisonemissing = empty($_POST[$fieldname]);
if ($thisonemissing) {
$missing .= $row['name'].', ';
}
$allthere = $allthere && !$thisonemissing;
}
}
$missing = substr($missing, 0, -2);
if ($allthere) {
$missing = '';
}
}
} else {
$missing = '';
}
- so the "$requried" var should store a list of all the arguments that are required. I added some test debug code to output this array, and it only contained one element = "1", which I'm assuming is the email address. further research is required
- also, I realized that there doesn't appear to be a per-list attribute config section, which makes me think that you can't have distinct attribute sets for different subscriber lists, which may be a problem. Further research into this is required
- ...
- I dug into the db further to see what I could dig up: I found only 1 row in the table 'phplist_user_attribute' with my 'termsofservice' attribute, which is indeed marked as required
MariaDB [phplist_db]> describe phplist_user_attribute; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | name | varchar(255) | NO | MUL | NULL | | | type | varchar(30) | YES | | NULL | | | listorder | int(11) | YES | | NULL | | | default_value | varchar(255) | YES | | NULL | | | required | tinyint(4) | YES | | NULL | | | tablename | varchar(255) | YES | | NULL | | +---------------+--------------+------+-----+---------+----------------+ 7 rows in set (0.00 sec) MariaDB [phplist_db]> select * from phplist_user_attribute; +----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+ | id | name | type | listorder | default_value | required | tablename | +----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+ | 4 | I agree to the OSE <a href=\'https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy\'>Privacy Policy</a>. | checkbox | 0 | | 1 | termsofservice | +----+-------------------------------------------------------------------------------------------------------------------------------+----------+-----------+---------------+----------+----------------+ 1 row in set (0.00 sec) MariaDB [phplist_db]>
- note there were two other tables with 'attribute' in their name, which appear to track & store the fact that each user has accepted the terms of service as required by gpdr
MariaDB [phplist_db]> select * from phplist_admin_attribute -> ; Empty set (0.00 sec) MariaDB [phplist_db]> select * from phplist_admin_attribute; Empty set (0.00 sec) MariaDB [phplist_db]> select * from phplist_user_user_attribute; +-------------+--------+-------+ | attributeid | userid | value | +-------------+--------+-------+ | 4 | 3 | on | | 4 | 6 | on | | 4 | 7 | on | | 4 | 8 | on | +-------------+--------+-------+ 4 rows in set (0.00 sec) MariaDB [phplist_db]>
- I submitted a bug report with all these details to the phplist mantis https://mantis.phplist.org/view.php?id=19503
- further testing should be done, modifying the script to ensure that it always populates the $required attributes array and checks to see if theyr'e missing
Mon Oct 29, 2018
- revisiting my ajax solution for phplist, I have 3 outstanding issues
- I need to harden the ACCESS_CONTROL_ALLOW_ORIGIN options and figure out how to set it up for mulitple domains
- I need to format it better since I removed the tables
- I need to get it to handle errors properly
- the last one, error catching, was discussed here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491
- the full code of the ajax form for error handling is given here https://github.com/scotch-io/ajax-forms-jquery/blob/master/magic.js
- an explination is found here https://scotch.io/tutorials/submitting-ajax-forms-with-jquery
- I tried to get some specific info, but all I got was a status code of 0 and a status text of 'error' — both when I give it proper & improper data.
Fri Oct 26, 2018
- I made some progress on the ajax form. did some cleanup & better functionality when JS is disabled
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html 100% 5009 49.5KB/s 00:00
user@ose:~$ cat phplistAjax.html
<html><body>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<noscript>
Please subscribe to our newsletter on our phplist site at <a href="https://phplist.opensourceecology.org/lists/index.php">https://phplist.opensourceecology.org/lists/</a>
</noscript>
<script type="text/javascript">
function checkform()
{
for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
if (document.subscribeform.elements[fieldstocheck[i]].checked) {
} else {
alert("The following field is required: "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
} else {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
}
}
for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
return false;
}
}
if (! checkEmail()) {
alert("Email address is not valid");
return false;
}
return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
fieldstocheck[fieldstocheck.length] = value;
fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
groupstocheck[groupstocheck.length] = value;
groupnames[groupnames.length] = name;
}
function compareEmail()
{
return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}
function checkEmail()
{
var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.subscribeform.elements["email"].value);
}
function checkGroup(name,value)
{
option = -1;
for (i=0;i<document.subscribeform.elements[name].length;i++) {
if (document.subscribeform.elements[name][i].checked) {
option = i;
}
}
if (option == -1) {
alert ("Please enter your "+value);
return false;
}
return true;
}
</script>
<script type="text/javascript">
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
async: true,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
//contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
</script>
<div id="phplistAjaxSubscribeFormWrapper" style="display:none;">
<form method="post" action="" name="subscribeform" id="subscribeform">
<div class="required"><label for="email">Email address *</label></div>
<input type=text name=email required="required" placeholder="" size="40" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
<input type="hidden" name="htmlemail" value="1" />
<input type="checkbox" name="attribute4" value="on" class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span>
<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script>
<input type="hidden" name="list[2]" value="signup" />
<input type="hidden" name="listname[2]" value="newsletter"/>
<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>
</form>
<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>
</div>
<script type="text/javascript">
// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';
</script>
</body></html>
user@ose:~$
- meeting with Marcin about phplist
- I explained GDPR and how Alex's site is not properly meeting our obligations
- I asked how we should integrate our phplist newsletter to social media, and Marcin said don't worry about it; we should just delete the links to like/retweet the newsletter posting for now. Perhaps in the future we can script a tool to automatically send a campaign when a new post is made to our wordpress site, then post to social media & add-back the "like" and "retweet" links then. But, for now, we just need to get him suited to send newsletters without jumping through 4 distinct gmail accounts.
- I asked about attributes. He said that we should import all existing lists' attributes into phplist. therefore, we're going to want to enumerate all of this info into our gdpr-compliant Privacy Policy
- I explained how phplist ajax & rest api works (or doesn't work) to add users to our subscription list without requring them to leave out other domain. I'm currently ironing-out final issues in the ajax solution.
- I explained how the repermission campaign would be requried after imports.
- we discussed how we would have multiple lists one for each of the existing lists, and then we would create one more list for the general "ose newsletter" which only needs the 3 basic & required attributes = timestamp, email address, & accepted PP.
Wed Oct 24, 2018
- continuing with my efforts to get a custom form to add a subscriber to our phplist site using AJAX..
- the source of offending modsecurity rule id=960010 is in /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
[root@hetzner2 ~]# grep '960010' -B 30 -A 2 /etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
# Restrict which content-types we accept.
#
# TODO Most applications support only two types for request bodies
# because that is all browsers know how to produce. If you are using
# automated tools to talk to the application you may be using other
# content types and would want to change the list of supported types.
#
# Note though that ModSecurity parses only three content types:
# application/x-www-form-urlencoded, multipart/form-data request and
# text/xml. The protection provided for any other type is inferior.
#
# TODO There are many applications that are not using multipart/form-data
# types (typically only used for file uploads). This content type
# can be disabled if not used.
#
# NOTE We allow any content type to be specified with GET or HEAD
# because some tools incorrectly supply content type information
# even when the body is not present. There is a rule further in
# the file to prevent GET and HEAD requests to have bodies to we're
# safe in that respect.
#
# NOTE Use of WebDAV requires "text/xml" content type.
#
# NOTE Philippe Bourcier (pbourcier AT citali DOT com) reports
# applications running on the PocketPC and AvantGo platforms use
# non-standard content types:
#
# M-Business iAnywhere application/x-mal-client-data
# UltraLite iAnywhere application/octet-stream
#
SecRule REQUEST_METHOD "!^(?:GET|HEAD|PROPFIND|OPTIONS)$" "phase:1,chain,t:none,block,msg:'Request content type is not allowed by policy',rev:'2',ver:'OWASP_CRS/2.2.9',maturity:'9',accuracy:'9',id:'960010',tag:'OWASP_CRS/POLICY/ENCODING_NOT_ALLOWED',tag:'WASCTC/WASC-20',tag:'OWASP_TOP_10/A1',tag:'OWASP_AppSensor/EE2',tag:'PCI/12.1',severity:'2',logdata:'%{matched_var}'"
SecRule REQUEST_HEADERS:Content-Type "^([^;\s]+)" "chain,capture"
SecRule TX:0 "!^%{tx.allowed_request_content_type}$" "t:none,ctl:forceRequestBodyVariable=On,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.critical_anomaly_score},setvar:tx.%{rule.id}-OWASP_CRS/POLICY/CONTENT_TYPE_NOT_ALLOWED-%{matched_var_name}=%{matched_var}"
[root@hetzner2 ~]#
- now, I am setting the enctype of the form to "multipart/form-data" in the ajax form, but modsecurity is complaining that it's "text/plain"
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
- I found a list of supported data types in /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf
[root@hetzner2 ~]# grep 'application/xml' /etc/httpd/modsecurity.d/* grep: /etc/httpd/modsecurity.d/activated_rules: Is a directory /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf: setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json', \ [root@hetzner2 ~]#
- I also found a note that the offending data type (text/plain) was indeed added to the acceptable list in the CRS 3.0 back in 2016; not sure why we don't have that https://github.com/SpiderLabs/owasp-modsecurity-crs/issues/208
- so I could update the CRS's config to include "text/plain", but I think the best solution is to fix the AJAX code to send the correct data type.
- the offending code is probably somewhere in this jquery.ajax() call (as obtained from the example zip in the phplist forum thread https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',
dataType: 'html',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
- I've never used jquery, but my guess is that we need to set the "dataType" here; it's documented here https://api.jquery.com/jQuery.ajax/
- oh, hmm, actualy reading the docs suggest that the 'dataType' defines what data type we expect _back_ from the server https://api.jquery.com/jQuery.ajax/
The type of data that you're expecting back from the server. If none is specified, jQuery will try to infer it based on the MIME type of the response (an XML MIME type will yield XML, in 1.4 JSON will yield a JavaScript object, in 1.4 script will execute the script, and anything else will be returned as a string). The available types (and the result passed as the first argument to your success callback) are:
- so I guess I need to dig into the jquery.serialize() command
<form method="post" action="" name="subscribeform" enctype="multipart/form-data">
...
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
...
}
- so the documentation on the serialize function explicitly states that it creates a string https://api.jquery.com/serialize/
- ah, so the option I want is "contentType" https://api.jquery.com/jQuery.ajax/
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize(); jQuery.ajax( {
type: 'POST',
data: data,
url: 'https://phplist.opensourceecology.org/lists/?p=subscribe&id=1',
dataType: 'html',
contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
}, error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
- there's a new modsecurity alert
[Wed Oct 24 20:43:47.876714 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Multipart parsing error (init): Multipart: Boundary not found in C-T. [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"] [Wed Oct 24 20:43:47.876765 2018] [:error] [pid 6907] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "REQBODY_ERROR" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "151"] [id "960912"] [rev "1"] [msg "Failed to parse request body."] [data "Multipart: Boundary not found in C-T."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "phplist.opensourceecology.org"] [uri "/lists/"] [unique_id "W9DZg4Q40X33JgAIGX84xgAAADM"]
- so this is probably because the content is--in fact--actually "text/plain" there's no multipart boundray since I just lied about what the mime type is. The solution should still be to get serialize() to output the proper mime type
- well, it appears that the alternative solution (as is often used for form data including binary content--such as images), is to use the FormData, but this is a browser-specific thing https://developer.mozilla.org/en-US/docs/Web/API/FormData
- so proceeding with FormData would likely limit the browsers in which this ajax form will work. It's really not requred for the small fields needed for subscriber attributes data in this form. so let's just ignore the modsecurity role
- well, I tried to overwrite the existing allowable mime types in our vhost file (making restore from backups easier). that didn't work
- I tried to just add the rule id = 960010 to the long list of SecRuleRemoveById list in the vhost file; that also didn't work
- the only way it worked as if I actually edited the list in the /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf file, adding "text/plain" to the list (followed by an apache reload) https://stackoverflow.com/questions/41604674/customizing-apache-mod-security-to-accept-content-type-text-plain
[root@hetzner2 modsecurity.d]# pwd /etc/httpd/modsecurity.d [root@hetzner2 modsecurity.d]# grep -C5 'text/plain' modsecurity_crs_10_config.conf SecAction \ "id:'900012', \ phase:1, \ t:none, \ setvar:'tx.allowed_methods=GET HEAD POST OPTIONS', \ setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json|text/plain', \ setvar:'tx.allowed_http_versions=HTTP/0.9 HTTP/1.0 HTTP/1.1', \ setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/', \ setvar:'tx.restricted_headers=/Proxy-Connection/ /Lock-Token/ /Content-Range/ /Translate/ /via/ /if/', \ nolog, \ pass" [root@hetzner2 modsecurity.d]#
- so this time it just got a 200 from the server (logs all look clean), but the user didn't get added. The js console shows some (unrelated?) errors about our jquery script file
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More] jquery.min.js:112:77 SyntaxError: missing ) after argument list[Learn More] phplistAjax.php:1:58 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/ jquery.min.js:127:343 TypeError: $(...).tagsinput is not a function[Learn More] phplistAjax.php:7:2 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2 ready https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:100:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371
- I checked out the sourcecode of our phplist subscribe page, and I discovered that it has its own checkform() js function, and some helper functions. I just copied these to my ajax form https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
- unrealted, but I saw that at the top of this sourcecode is a php command that for some reason came out as text, not processed by php
<head> <meta http-equiv="pragma" content="no-cache" /> <meta http-equiv="Cache-Control" content="no-cache, must-revalidate" /> <meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" /> <meta name="Author" content="Michiel Dethmers - http://www.phplist.com" /> <meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" /> <meta name="Powered-By" content="phpList version 3.3.3" /> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <link rel="SHORTCUT ICON" href="./images/phplist.ico" /> <title>Subscribe to our Newsletters</title><meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" /> <meta name="theme-color" content="#2C2C2C"/> <link rel="apple-touch-icon" href="./images/phplist-touch-icon.png" /> <link rel="apple-touch-icon-precomposed" href="./images/phplist-touch-icon.png" /> <link rel="stylesheet" href="admin/ui/phplist-ui-bootlist/css/style.css?v=<?php echo filemtime(dirname(FILE).'/css/style.css'); ?>" /> </head>
- it looks like this comes from pagetop_minified.php
[root@hetzner2 lists]# pwd
/var/www/html/phplist.opensourceecology.org/public_html/lists
[root@hetzner2 lists]# cat admin/ui/dressprow/pagetop_minified.php
<?php
/*
We request you retain the full headers below including the links.
This not only gives respect to the large amount of time given freely
by the developers, but also helps build interest, traffic and use of
phpList, which is beneficial to it's future development.
Michiel Dethmers, phpList Ltd 2003 - 2015
*/
?>
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 TransitionalEN" "http:www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" >
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="<?php echo $_SESSION['adminlanguage']['iso']?>" lang="<?php echo $_SESSION['adminlanguage']['iso']?>" dir="<?php echo $_SESSION['adminlanguage']['dir']?>">
<head>
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Cache-Control" content="no-cache, must-revalidate" />
<meta name="License" content="GNU Affero General Public License, http://www.gnu.org/licenses/agpl.html" />
<meta name="Author" content="Michiel Dethmers - http://www.phplist.com" />
<meta name="Copyright" content="Michiel Dethmers, phpList Ltd - http://phplist.com" />
<meta name="Powered-By" content="phplist version <?php echo VERSION?>" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="SHORTCUT ICON" id="favicon" href="./images/phplist.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<link rel="stylesheet" href="ui/dressprow/css/all.min.css?v=<?php echo filemtime(dirname(FILE).'/css/all.min.css'); ?>" />
<?php
if (isset($GLOBALS['config']['head'])) {
foreach ($GLOBALS['config']['head'] as $sHtml) {
echo $sHtml;
echo "\n";
echo "\n";
}
}
[root@hetzner2 lists]#
- interstingly, there's php lines above it (ie: the "phplist version") execute without issues
- this is somewhat related, but not really https://mantis.phplist.org/view.php?id=18447
- ah, this is it https://mantis.phplist.org/view.php?id=19307
- looks like the fix was to just remove it https://github.com/phpList/phplist-ui-bootlist/pull/62
- anyway, it looks like the error "SyntaxError: missing ) after argument list" was caused by my attribute with a quote in it. Xheni warned me about this here https://mantis.phplist.org/view.php?id=19436
- I replaced my links' encapsulating double qoutes with single quotes as a temp fix
- ok, better fix, update the attribute to use single quotes in phplist's admin ui
- so actually the issue is quotes, not slashes.
- ok, that elminated the error entirely. Now I just have
Loading mixed (insecure) display content “http://powered.phplist.com/images/3.3.3/power-phplist.png” on a secure page[Learn More] jquery.min.js:112:77 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/ jquery.min.js:127:343 TypeError: $(...).tagsinput is not a function[Learn More]
- this is frustrating, but I confirmed that signup *does* work from our actual subscribe page https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1
- I noticed that there were a few js files included in the phplist site's code, so I added these to my ajax form
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script> <script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script> <script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
- now I get a pop-up alert = "Sorry, we were unable to process your subscription."
- someone else posted about this issue a couple years ago https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
- ah, another point. I have the URL wrong! per https://discuss.phplist.org/t/ajax-subscribe-api/974/15
url: 'https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=1',
- but, in fact, it should use 'asubscribe' intead of 'subscribe' = https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=1
- per the thread above, I also changed it to default to html & not confirm email address
- it's still not working. this is taking for fucking ever. I gotta get some POC. I'll simplify it by stripping out the "I agree" checkbox for now. If I can fucking get that to work, I'll try to add a simple checkbox and work from there..
- I created a new list and cooresponding subscribe page that's as simple as possible: it's just an email address https://phplist.opensourceecology.org/lists/index.php?p=subscribe&id=2
- ugh, even that won't work.
- oh, I got it to work! I had to set both the name and the id attributes of the form field to "subscribeform". Note that the form coming from phplist's html only sets the 'name' attribute, but the jquery serialize() function is called against the 'id' attribute. otherwise, it probably just sent an empty string to the subscribe page!
<form method="post" action="" name="subscribeform" id="subscribeform">
- woohoo! I switched back to the list=1 (with the checkbox agreeing to our PP), and it worked! When the checkbox is *not* ticked, a JS pop-up yells at you:
The following field is required: I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.
- unfortunately, it still says that it fails even when it succeeds, as the thread above pointed out here https://discuss.phplist.org/t/having-problems-with-ajax-subscription-form-from-seperate-server/1491/3
- fwiw, this is what woked today. Tomorrow, I should look into making this visible only if JS is enabled. Otherwise, I should just display an anchor link the user to our phplist subscription page
user@ose:~$ scp opensourceecology.org:phplistAjax.html .
phplistAjax.html 100% 5329 53.4KB/s 00:00
user@ose:~$ cat phplistAjax.html
<html><body>
<head>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/js/phplistapp.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>
<script type="text/javascript">
function checkform()
{
for (i=0;i<fieldstocheck.length;i++) {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
if (document.subscribeform.elements[fieldstocheck[i]].checked) {
} else {
alert("The following field is required: "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
} else {
if (eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].value") == "") {
alert("Please enter your "+fieldnames[i]);
eval("document.subscribeform.elements['"+fieldstocheck[i]+"'].focus()");
return false;
}
}
}
for (i=0;i<groupstocheck.length;i++) {
if (!checkGroup(groupstocheck[i],groupnames[i])) {
return false;
}
}
if (! checkEmail()) {
alert("Email address is not valid");
return false;
}
return true;
}
var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name)
{
fieldstocheck[fieldstocheck.length] = value;
fieldnames[fieldnames.length] = name;
}
var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name)
{
groupstocheck[groupstocheck.length] = value;
groupnames[groupnames.length] = name;
}
function compareEmail()
{
return (document.subscribeform.elements["email"].value == document.subscribeform.elements["emailconfirm"].value);
}
function checkEmail()
{
var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
return re.test(document.subscribeform.elements["email"].value);
}
function checkGroup(name,value)
{
option = -1;
for (i=0;i<document.subscribeform.elements[name].length;i++) {
if (document.subscribeform.elements[name][i].checked) {
option = i;
}
}
if (option == -1) {
alert ("Please enter your "+value);
return false;
}
return true;
}
</script>
<script type="text/javascript">
function submitForm() {
successMessage = 'Thank you for your registration. Please check your email to confirm.';
data = jQuery('#subscribeform').serialize();
jQuery.ajax( {
type: 'POST',
data: data,
async: true,
url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=2',
dataType: 'html',
//contentType: 'multipart/form-data',
success: function (data, status, request) {
jQuery("#result").empty().append(data != '' ? data : successMessage);
jQuery('#attribute1').val('');
jQuery('#email').val('');
},
error: function (request, status, error) { alert('Sorry, we were unable to process your subscription.'); }
});
}
</script>
</head>
<form method="post" action="" name="subscribeform" id="subscribeform"><table border=0>
<tr><td><div class="required"><label for="email">Email address *</label></div></td>
<td class="attributeinput"><input type=text name=email required="required" placeholder="" size="40" id="email" />
<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script></td></tr><input type="hidden" name="htmlemail" value="1" />
<tr><td colspan="2">
<input type="checkbox" name="attribute4" value="on" class="attributeinput" id="attribute4" />
<span class="required"><label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>. *</label></span><script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy'>Privacy Policy</a>.");</script></td></tr>
</table><input type="hidden" name="list[2]" value="signup" /><input type="hidden" name="listname[2]" value="newsletter"/><div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div><input type=submit name="subscribe" value="Subscribe to the selected newsletters" onClick="return checkform();"> <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website" ><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0" /></a></p></div>
<button class='button'
onclick="if (checkform()) {submitForm();} return false;"
>Subscribe</button>
<div id="result" style="color: red;"></div>
</div>
</body></html>
user@ose:~$
Tue Oct 23, 2018
- Sam got back to me about integrating a signup list into wordpress via AJAX or the REST API. They said the API doesn't yet support attributes (that conflicts with the example I saw yesterday, though). Sam said adding that feature to the API will come with time, but we could fund it to be built in 6-8 weeks for ~$4,500. Of course, we could also just develop it ourselves and PR it into their repo (it uses Symfony and Doctrine) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/4
- Sam didn't give a direct answer, but suggested that it's probably possible to do with ajax
- fuck it; let's see if I can manually create a form that does what I want with ajax. I'll use this old example as a template https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
- I put the file up on the microfactory site in the wp-content dir.
- I got a success message (Thank you for your registration. Please check your email to confirm.), but it didn't add to the subscribers list. I checked the html console, and I got an error due to Cross-Origin
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://phplist.opensourceecology.org/lists/?p=subscribe&id=1. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘http://phplist.opensourceecology.org’).
- I found that phplist inclues logic to define the "Acces-Control-Allow-Origin" header based on a config file option
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'Allow-Origin' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
[root@hetzner2 phplist.opensourceecology.org]# grep -ir 'ACCESS_CONTROL_ALLOW_ORIGIN' *
public_html/lists/index.php:header('Access-Control-Allow-Origin: '.ACCESS_CONTROL_ALLOW_ORIGIN);
public_html/lists/admin/init.php:if (!defined('ACCESS_CONTROL_ALLOW_ORIGIN')) {
public_html/lists/admin/init.php: define('ACCESS_CONTROL_ALLOW_ORIGIN', $GLOBALS['scheme'].'://'.$_SERVER['HTTP_HOST']);
[root@hetzner2 phplist.opensourceecology.org]#
- and here's the relevant option in the config file https://resources.phplist.com/system/config/access_control_allow_origin
- apparently the protcol (https://) must be specified. I finally got a new error after adding this line to the phplist config.php file
// allow AJAX queries to add subscribers to our db from other domains
define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://microfactory.opensourceecology.org" );
- now the error is
Blocked loading mixed display content “http://powered.phplist.com/images/3.3.3/power-phplist.png%E2%80%9D[Learn More] jquery.min.js:112:77 SyntaxError: missing ) after argument list[Learn More] phplistAjax.php:1:58 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange https://microfactory.opensourceecology.org/wp-content/jquery.min.js:129:371 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help http://xhr.spec.whatwg.org/ jquery.min.js:127:343 TypeError: $(...).tagsinput is not a function[Learn More] phplistAjax.php:7:2 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:7:2 ready https://microfactory.opensourceecology.org/wp-content/jquery.min.js:24:208 <anonymous> https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:3:1 globalEval https://microfactory.opensourceecology.org/wp-content/jquery.min.js:29:285 Qa https://microfactory.opensourceecology.org/wp-content/jquery.min.js:16:201 each https://microfactory.opensourceecology.org/wp-content/jquery.min.js:30:149 domManip https://microfactory.opensourceecology.org/wp-content/jquery.min.js:110:185 append https://microfactory.opensourceecology.org/wp-content/jquery.min.js:103:419 success https://microfactory.opensourceecology.org/wp-content/phplistAjax.php:57:13 b https://microfactory.opensourceecology.org/wp-content/jquery.min.js:124:1 ajax/x.onreadystatechange
- this time jquery just fucking fetched the whole form and put it on the page somehow.
<div class="panel"> <br> <div class="content well"><h3>Subscribe to our newsletters</h3> <div class="error"><span class="required">* required fields are marked red</span></div> <div class="error missing">The following required values are missing: Email addresses you entered do not match</div><br> <form method="post" action="" name="subscribeform"><table border="0"> <tbody><tr><td><div class="required"><label for="email">Email address *</label></div></td> <td class="attributeinput"><input name="email" required="required" placeholder="testa@example.com" size="40" id="email" type="text"> </td></tr> <tr><td><div class="required"><label for="confirm">Confirm your email address *</label></div></td> <td class="attributeinput"><input name="emailconfirm" required="required" value="" size="40" id="confirm" type="text"> </td></tr><tr><td colspan="2"> <span class="attributeinput"> <input name="textemail" value="1" id="textemail" type="checkbox"></span> <span class="attributename"><label for="textemail">I prefer to receive emails in Text format</label></span> </td></tr> <tr><td colspan="2"> <input name="attribute4" value="on" class="attributeinput" id="attribute4" type="checkbox"> <span class="required"><label for="attribute4">I agree to the OSE <a href="https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy">Privacy Policy</a>. *</label></span></td></tr> </tbody></table><input name="list[2]" value="signup" type="hidden"><input name="listname[2]" value="newsletter" type="hidden"><div style="display:none"><input name="VerificationCodeX" value="" size="20" type="text"></div><input name="subscribe" value="Subscribe to the selected newsletters" onclick="return checkform();" type="submit"> <a href="http://phplist.opensourceecology.org/lists/?p=unsubscribe&id=1">Unsubscribe</a></form><p class="poweredby" style="text-align:center"><a href="https://www.phplist.com/poweredby?utm_source=pl3.3.3&utm_medium=poweredhostedimg&utm_campaign=phpList" title="visit the phpList website"><img src="http://powered.phplist.com/images/3.3.3/power-phplist.png" title="powered by phpList version 3.3.3, © phpList ltd" alt="powered by phpList 3.3.3, © phpList ltd" border="0"></a></p></div> </div>
- I used this html form data to merge into the example one
- I hit a modsecurity issue with rule #960010
Request content type is not allowed by policy
- I think it may be related to the content type submitted by the form somewhere defaulting to "text/plain" *shrug*
- I had lost my password for our phplist install due to my ssd crash; I reset it (noting that you cannot login with your email address; you must use your username--as addressed in the password reset email. In my case, 'maltfield').
- I also added an admin for marcin
- I confirmed that the shared 'admin' user is in our shared ose keepass db
Mon Oct 22, 2018
- Marcin sent me our other 3 email spreadsheets; I dug through them to see what attributes we collected
- according to phplist plugin page, the rest api doesn't support setting attributes https://resources.phplist.com/plugin/restapi
- however, there does appear to be support for non-email attributes per this document, even though it does mention on the main page of this github repo that it's deprecated (the replacement for php4 isn't stable, however) https://github.com/phpList/phplist-plugin-restapi/blob/master/plugins/restapi2/docs/api.apib#L185-189
- the replacement api's repo does say that it can work for php3 https://github.com/phpList/rest-api
This new REST API can also be used to provide REST access to an existing phpList 3 installation. For this, the phpList 3 installation and the phpList 4 installation with the REST API need to share the same database. For security reasons, the REST APIs from phpList 3 and phpList 4 should not be used for the same database in parallel, though.
Sat Oct 20, 2018
- I did a search for wordpress plugins using the keyword "phplist". Here's some noteable results
- Sign Me Up https://wordpress.org/plugins/sign-me-up/
- - last updated 3 years ago
- + explicitly lists that it connects with phplist using AJAX
- - says that it doesn't support additional attributes (email address only). This will probably not work as we need to at least have a checkbox for the user to accept our gdpr-ready Privacy Policy
- there's a lot of info about this plugin here https://www.jaromy.net/wordpress-plugins/sign-me-up[[1]]
- WP PHPList https://wordpress.org/plugins/phplist-form-integration/
- - last updated 9 years ago
- yPHPlista https://wordpress.org/plugins/yphplista/
- - last updated 8 years ago https://wordpress.org/plugins/yphplista/
- Sign Me Up https://wordpress.org/plugins/sign-me-up/
- I tried searching the phplist forums for "wordpress" to dig through recent discussions about integrating phplist with wordpress
- I found this interesting discussion hinting to having a phplist plugin automatically send email campaigns after a new wordpress post is in. I really like this idea (ie: RSS scraping -> email campaign). Prior to sending the email campaign, we could post a link to it on facebook, twitter, etc. Then we could add a link to "like" or "retweet" the post (on social media) to the original post (on wordpress) within the email template. https://discuss.phplist.org/t/how-is-content-generated-in-wordpress/1656/6
- so 6 months ago someone asked my question: is there an official wp plugin for phplist? Sam Tuke (CEO of phplist) responded stating that there isn't but it's needed. And expounded on how the AJAX solution is the old way and that the future phplist wp plugin would work via the REST API (I like this much better too) https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894
- this may work with the AJAX way https://discuss.phplist.org/t/wordpress-integration-newsletter-sign-up-plugin/1723/15
- And there was this old post for 2016 about the 9-year-old "WP PHPList" (phplist-form-integration) wordpress plugin https://discuss.phplist.org/t/phplist-wordpress-integration-solved/2279/2
- I lost my password for phplist's discourse site due to my ssd crash, so I generated a new one
- Interestingly, it looks like phplist's discourse site is SSO using wordpress as the account backend
- I replied to this post, asking Sam if there's an option for sending our new subscriber's attributes to the stable version phplist via either AJAX or REST https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/3
Fri Oct 19, 2018
- I'm back in NYC after restoring my laptop following my ssd crash
- sent Sara information about wp-cli and managing wordpress
- I also sent an email to Marcin asking if he authorizes them having ssh and/or shraed ose keepass access
- specifically, the plugin that Sara wanted to install was this https://wordpress.org/plugins/custom-css-js
- the above plugin is freemium. the reviews suggested this free alterantive for more features, but it's a less popular plugin https://wordpress.org/plugins/scripts-n-styles/
- Marcin sent me a confusing email about utilization of google analytics. One of our first conversations with Marcin back in 2017 was about how OSE was working to migrate _away_ from Google services. We've since stripped analytics from all our sites. We looked at Piwik, but--due to security issues--we decided to go with awstats.
- I restated that we use awstats, and asked what are the benefits of using google analytics
- I added the microfactory site to our awstast generation cron job, and confirmed that I could view all the stats so far this month https://awstats.opensourceecology.org:4443/awstats.microfactory.opensourceecology.org.html
- In general, I'm very confused about where the line is drawn between giving up on FLOSS and moving to some closed-source SaaS solution here at OSE.
- I've been hacking the hell out of phplist so that we can use an ugly FLOSS solution, but Mailchimp would be much better, and very cheap for our needs.
- We still don't have a decent issue tracker. Mantis is a long-standing TODO, but we could use Jira for free because we're a nonprofit.
- Google Analytics is surely more powerful thatn awstats, but does that really justify the use of closed-source tools that exploit our visitor's data? Why draw the line here between awstats & GA but not use Jira or Mailchimp like most other nonprofits?
- Marcin sent me an error when trying to post our email thread about installing a wordpress plugin to the wiki
- fixed by modsec whitelist
- 950019, generic attack = email injection
- fixed by modsec whitelist
- I updated the documentation on the wiki for the quick-start guide to configuring 2FA for our wordpress sites using the Google Authenticator plugin https://wiki.opensourceecology.org/wiki/2FA#Quick_Start_Guide
- I added a step to enable relaxed mode, so the user's phone only has to be correct within 4 minutes, rather than 30 seconds.
- I recommended use of the oandbackup app, which is the FLOSS replacement for Titanium Backup https://f-droid.org/en/packages/dk.jens.backup/
- ...
- began to revisit my install of the "Newsletter Sign-Up" wordpress plugin on the fef site
- I logged into the fef wordpress ite
- I navigated to the Plugins -> "Newsletter Sign-Up" -> "Settings" link (note that this redirects to the main panel nav bar's "Newsl. Sign-Up" link.
- I changed "Select your mailinglist provider" to "PHPList"
- our subscriber link is this, so I left the "PHPList list ID" at "1" https://phplist.opensourceecology.org/lists/?p=subscribe&id=1
- I set the "Newsletter form action" to "https://phplist.opensourceecology.org/lists/?p=subscribe&id=1"
- this plugin appears to be mostly built around the trick of having someone signup when leaving a comment. That's a feature that we may want to disable. Mostly we want a widget and a form embedded into a post/page.
- I picked a bad site, fef, for testing widgets. it doesn't use them!
- I tested adding a form for signing up to the phplist newsletter to the lola page by adding "[nsu_form]" to the content of the post https://fef.opensourceecology.org/2015/09/03/lola/
- when I submitted on this form, it just fucking redirected me to the phplist site. it didn't even auto-fill the form! what rubbish! what we want here is for a seamless submittion to the phplist site without the user having to leave the site that they are on.
- ugh, I just realized that the "Newsletter Sign-Up" form's main plugin page on the wp site shows a message indicating that the plugin is no longer maintained (it was last updated 11 months ago) https://wordpress.org/plugins/newsletter-sign-up/
- I deactivated the "Newsletter Sign-Up" plugin and deleted the related content & widgets from the fef site.
- looks like I lost my logs from when I discovered the "Newsletter Sign-Up" and similar wordpress plugins
- I dug up some options
- https://wordpress.org/plugins/sign-me-up/
- - last updated 3 years ago
- https://wordpress.org/plugins/yphplista/
- - last updated 8 years ago
- https://wordpress.org/plugins/sign-me-up/
- it looks like phplist.com (not .org) offers a solution to this issue, the so-called "AJAX" solution so subscribres can signup "...and stay on your website" https://www.phplist.com/ajaxdemo
- found some info on enabling this the '.org' way. note that there may be issues with the ACCESS_CONTROL_ALLOW_ORIGIN https://discuss.phplist.org/t/ajax-subscribe-api/974/7
define('ACCESS_CONTROL_ALLOW_ORIGIN', 'www.mydomain.de');
define('PUBLIC_PROTOCOL','https');
- I found a few resources about ajax on phplist.com's wiki https://resources.phplist.com/start?do=search&id=ajax
- ugh, this just references the 2x no-longer-maintained plugins I found above https://resources.phplist.com/develop/wordpress?s[]=ajax
- there's only 1x reference to "ajax" in the phplist.org manual it referes to the Ethical Pet case study. Yeah, the one that migrated off from phplist..
- important, but a fucking stub. can we get an example listing 3x domains? https://resources.phplist.com/system/config/access_control_allow_origin?s[]=ajax
- the discuss thread above includes this example file from subscription_form_example.html (in subscription_form_example.html.zip) https://discuss.phplist.org/uploads/default/original/1X/491a212528c7d5a4297d5449294fb5300778712c.zip
- ugh, the first line includes the jquery src @ googleapis.com. Let's not make our users require decentraleyes to prevent google from tracking them on our site..
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
- it's looking like the best option is to use the no longer maintained "Newsletter Sign-Up" wp plugin still :\ I re-enabled it and added the line "[nsu_form]" to the lola paga again. I'll try to debug that before reinventing the wheel https://fef.opensourceecology.org/2015/09/03/lola/
- I checked the code; there's not even any JS; it's just a fucking form with submit action to our distinct phplist site (I figured it was some ajax cross-site rejection failure causing fall-back to the redirect). Maybe this plugin isn't what we want still..I disabled it again.
Sun Oct 14, 2018
- bought & installed my replacement ssd after my old one died unexpectedly
- install os, restore backups, etc
- catching up on old email
Mon Oct 01, 2018
- Note that my logs for the first few weeks of this quarter were lost when my laptop's ssd crashed. Please mind the gap.