Phplist

From Open Source Ecology
Jump to: navigation, search

Introduction

Phplist is open source email list software. When OSE adopted it in 2018 for the OSE Newsletter, it was determined to be the most feature-rich FLOSS alternative to the gold-standard paid alternative, MailChimp.

Approving Members

  • Many members seem not to confirm their account.
  • Go to user, click on their name.
  • Under 'Is this subscriber confirmed (1/0)' - type 1. But only if the next line says that they are not blacklisted (they apparently unsubscribed)

Posting

To post as an admin, go to https://phplist.opensourceecology.org/lists/admin . Admin is found at /lists/admin

Transition Announcement

See:

Usage Notes

https://phplist.opensourceecology.org/lists/admin is the login screen. and admin at /lists/admin

  1. Next - Ajax form for embedding email collection, privacy policy, import lists
  1. GDPR - what we store (time stamp, check box, email), how long, includes privacy policy, we have to tell people what we store, why, do we share it, and accepts delete requests, and update) - ETA - mid-Nov new emails. Then repermission campaign.
  1. Simple sending of email with links. Period. 2000 emails per month
  2. Select multiple lists, and distinct attributes for each list
  3. Alex's website - violates GDPR
  4. Subscription for email list for website -
  5. Automated processes for later
  6. Email is collected and get a checkmark. Repermission campaign - click a link before they can receive the info.
  7. Privacy policy of Mediawiki - not GDPR compliant
  8. We store: email, timestamp, that they checked they saw our privacy policy
  9. Workshop Email List - update, rectify, delete, etc. We need to be able to remove a user.

User Experience

May 11, 2019 - MJ

Good job on the installl. User experience was quite good. Process from end to end was quite seamless. Sending frequent test messages is useful, and arrives in my inbox almost instantly. On our 200MB fiber connection, responsivity of phpList to saves and next steps was fast, almost instant. Going through all the steps was fast. Right clicking on images to resize them is a useful tool. Inserting images from URLs is a good idea. The only snag was an error upon one save - semething like cannot access server. But that was because one of my links had a bad link format, so the error actually forced me to validate links. After all links were corrected, the draft worked. All together, awesome. The only thing is - I did not receive the email myself. I should be on the True Fans and OSEmail lists.

Installation

Installation for phplist is relatively trivial. Their documentation leaves much to be desired, but it's fairly straight-forward.

libsodium

Note that, while not strictly a requirement on their website, phplist does effectively require libsodium and the corresponding pecl libsodium php extension to be installed on our OSE Server.

Without it, the random_compat library will throw an exception (which, by default, phplist will suppress from being written to the error logs): 

[Thu Aug 23 00:06:29.560157 2018] [:error] [pid 17617] [client 127.0.0.1:51262] PHP Fatal error:  Uncaught exception 'Exception' with message 'There is no suitable CSPRNG installed on your system' in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/inc/random_compat/random.php:204\nStack trace:\n#0 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/defaultconfig.php(3): random_bytes(10)\n#1 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php(103): require_once('/var/www/html/p...')\n#2 {main}\n  thrown in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/inc/random_compat/random.php on line 204

This, in part, is due to the use of 'open_basedir' in our /etc/php.ini file to whitelist the directories in which php is permitted to execute. The official response from the maintainers (see issue #99 of the random_compat repo on github [1]) is to simply add '/dev/urandom' to 'open_basedir' in '/etc/php.ini'. That didn’t seem like the wisest option, so--instead--we just install libsodium. Indeed, libsodium is the preferred source of entropy for random_compat, anyway [2]. 

yum install php-pecl-libsodium
httpd -t && service httpd restart

For more information about this issue and its fix, see

Proper File/Directory Ownership & Permissions

This section will describe how the file permissions should be set on an OSE phplist site.

For the purposes of this documentation, let's assume:

  1. vhost dir = /var/www/html/phplist.opensourceecology.org

Then the ideal permissions are:

  1. Files containing passwords (ie: config.php) should be located outside the docroot with not-apache:apache-admins 0040
  2. Files in the public_html/uploadimages dir should be www-data:www-data 0660
  3. All other files in the vhost dir should be not-apache:www-data 0040
  4. All other directories in the vhost dir should be not-apache:www-data 0050

This is achievable with the following idempotent commands: 

phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"

for vhost_dir in $phplist_sites; do
 
	for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
	for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
	for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
 
	for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
	for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
	for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
	for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done

done

Such that:

  1. the 'not-apache' user is a new user that doesn't run any software (ie: a daemon such as a web server) and whose shell is "/sbin/nologin" and home is "/dev/null".
  2. the www-data user is in the apache-admins group
  3. the www-data user is in the www-data group
  4. any human users that need read-only access to the phplist vhost files for debugging purposes and/or write access to the 'public_html/uploadimages/' directory (ie: to upload large files that are too large to be handled by the web servers chain), then that user should be added to the 'www-data' group
  5. any human users that need read-only access to the phplist vhost files, including config files containing passwords (ie: config.php), should be added to the 'apache-admins' group
  6. for anyone to make changes to any files in the docroot (other than 'public_html/uploadimages/'), they must be the root user. I think this is fair if they don't have the skills necessary to become root, they probably shouldn't modify the phplist core files anyway.

Why?

The following explains why the above permissions are ideal:

  1. All of the files & directories that don't need write permissions should not have write permissions. That's every file in a phplist docroot except the folder "public_html/uploadimages/" and its subfiles/dirs.
  2. World permissions (not-user && not-group) for all files & directories inside the docroot (and including the docroot dir itself!) should be set to 0 for all files & all directories.
  3. Excluding 'public_html/uploadimages/', these files should also not be owned by the user that runs a webserver (in cent, that's the 'www-data' user). For even if the file is set to '0400', but it's owned by the 'www-data' user, the 'www-data' user can ignore the permissions & write to it anyway. We don't want the www-data user (which runs the apache process) to be able to modify files. If it could, then a compromised webserver could modify a php file and effectively do a remote code execution.
  4. Excluding 'public_html/uploadimages/', all directories in the docroot (including the docroot dir itself!) should be owned by a group that contains the user that runs our webserver (in cent, that's the www-data user). The permissions for this group must be not include write access for files or directories. For even if a file is set to '0040', but the containing directory is '0060', any user in the group that owns the directory can delete the existing file and replace it with a new file, effectively ignoring the read-only permission set for the file.

Configuration

This section will guide the reader to finding the relevant files that properly configure phplist on our server. Note that we try to avoid listing the actual contents of config files in this document itself will likely be out-of-sync (rot) from the actual config files over time. The source-of-truth for the config files should be the server itself or the contents of the backups.

Important Files & Directories

The following files/directories are related to the phplist server and its config

  1. /etc/nginx/conf.d/phplist.opensourceecology.org.conf nginx tls-terminator frontend
  2. /etc/varnish/sites-enabled/phplist.opensourceecology.org varnish cache site-specific config file
  3. /etc/varnish/all-vhosts.vcl varnish cache config file enabling site-specific configs
  4. /etc/httpd/conf.d/00-phplist.opensourceecology.org.conf apache backend config file
  5. /var/www/html/phplist.opensourceecology.org/ phplist vhost files
  6. /var/www/html/phplist.opensourceecology.org/public_html/ phplist vhost document root
  7. /var/www/html/phplist.opensourceecology.org/config.php phplist config file
  8. /etc/cron.d/phplist phplist cron jobs
  9. /var/log/phplist/*.log log files for cron job scripts
  10. /etc/logrotate.d/phplist logrotate rules for phplist log files

Cron Jobs

Some phplist tasks are automated by scripts executed at regular intervals using cron jobs. For example, processing the queue and processing bounces.

Cross-Origin Resource Sharing

In order to allow external domains from POSTing data to our phplist web server, our phplist server must whitelist the external domain using the Access-Control-Allow-Origin header.

Phplist has built-in functionality for setting this header via the ACCESS_CONTROL_ALLOW_ORIGIN constant, but this header does not support defining multiple domains. This is an issue, for example, if we want to have an ajax form for registering users to our phplist newsletter from multiple distinct domains/websites (ie: www.opensourceecology.org and microfactory.opensourceecology.org). The solution recommended by the w3c is to "generate the Access-Control-Allow-Origin header dynamically" [3]. We implement this logic in nginx, as it's best to do it before the vanish cache.

Note the relevant comments in the phplist config file 

[root@hetzner2 ~]# grep -B8 "define('ACCESS_CONTROL_ALLOW_ORIGIN" /var/www/html/phplist.opensourceecology.org/config.php  

// allow AJAX queries to add subscribers to our db from other domains
// Note: The ACCESS_CONTROL_ALLOW_ORIGIN header does not support multiple
//       domains, so we instead have to maintain a whitelist logically and
//       dynamically return the relevant domain iff it's in the whitelist.
//       Therefore, we actually override this phplist ACCESS_CONTROL_ALLOW_ORIGIN
//       header in our nginx config. See the relevant nginx config file:
//         * /etc/nginx/conf.d/phplist.opensourceecology.org.conf
#define('ACCESS_CONTROL_ALLOW_ORIGIN', "https://www.opensourceecology.org" );
[root@hetzner2 ~]#

And the actual logic in the nginx config file 

[root@hetzner2 ~]# grep -A14 'location /' /etc/nginx/conf.d/phplist.opensourceecology.org.conf 

  location / {
    proxy_pass http://127.0.0.1:6081;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port 443;
    proxy_set_header Host $host;

    # handle cors whitelist for ajax subscription to phplist
    proxy_hide_header Access-Control-Allow-Origin;
    if ( $http_origin ~ "^https://(www.opensourceecology.org|microfactory.opensourceecology.org)$" ) {
      add_header Access-Control-Allow-Origin $http_origin;
    }

  }

Creating a New Signup Form

> collect emails in exchange for a free e-book?

iirc, my understanding is that is a violation of GDPR. I don't think that you can make PII collection a barrier for something free. What you

  • might* be able to do is prompt them with an interstitial page for their

email address, which would include a link below the form/submit button to "skip subscription and download now" or something. Then you're not making it a requirement, but you'll still probably get a lot of subscriptions.

As for creating a distinct list, you should be able to log into the admin panel and choose to create a new list. Unfortunately, I don't think there's an easy way to set it all up: I recommend just opening an existing list in one tab and the new list in another tab, and just swap back and forth copying the attributes from the old to the new as needed. Then add new attributes as needed.

You'll also might want to create a new subscribe page, which is distinct from the concept of the list itself. See the phpList docs: 

*

https://www.phplist.org/manual/books/phplist-manual/page/creating-your-lists 

*

https://www.phplist.org/manual/books/phplist-manual/page/creating-a-subscribe-page

Let me know if you get stuck somewhere.

Ajax Form

People can subscribe to our newsletters directly from our phplist virtual host server page's list page, but that is less than ideal. It is preferred to allow users to subscribe to our newsletter directly from our existing websites, such as our wiki or one of our wordpress blogs -- without having to navigate to a distinct website in order to register.

Phplist is developing a REST API, but--at the time of writing (Jan 2019)--it does not support attributes[4]. Therefore, we cannot send the value of the checkbox for the user accepting our Terms of Service through the API--a critical field for GDPR. Effectively then, the only way to achieve a newsletter registration form directly on one of our websites is to use the ajax form. This is not well documented, but an example ajax form for phplist is discussed on their forums here.

Example Form for OSEmail

A functional, commented, and robust example of an ajax form for users to subscribe to the OSEmail newsletter can be found on our osemain wordpress blog post Moving to Open Source Email List Software. Note that this supports error handling and provides a link to the subscribe page if the user has javascript disabled.

In order for the below snippet to be pasted into a wordpress post without wordpress mangling the javascript code with paragraph tags (<p>...</p>), it was necessary to disable the wpautop filter in wordpress. To do so for a single post, I added a very simple wordpress plugin, wpautop-control -- which allows the wpautop filter to be disabled by defining a custom field named "wpautop" with the value "false" on a per-post basis. Note that, in order to define a custom field for a given post, the "Custom Fields" checkbox must be enabled for the screen, which is configurable by clicking "Screen Options" at the top of the page when editing the post in wordpress. 

<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="https://phplist.opensourceecology.org/lists/admin/ui/phplist-ui-bootlist/js/dist/phpList_ui_bootlist.min.js"></script>

<noscript>
<div id="phplistAjaxSubscribeFormWrapper" style="background-color:#e4e8eb; padding:20px">
<br/>
<h3>Sign up for the OSEmail newsletter:</h3>
<p>
To subscribe to our OSEmail newsletter, please visit our phplist site at <a href="https://phplist.opensourceecology.org/lists/?p=subscribe&id=3">https://phplist.opensourceecology.org/lists/?p=subscribe&id=3</a>
</p>
</div>
</noscript>

<script type="text/javascript">
function checkform() {

	// first, clear the response div from the previous attempts results
	jQuery("#phplistResult").empty();

	for (i=0;i<fieldstocheck.length;i++) {
		if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].type") == "checkbox") {
			if (document.phplistSubscribeForm.elements[fieldstocheck[i]].checked) {
			} else {
				jQuery("#phplistResult").empty();
				alert("The following field is required:  "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");
				return false;
			}
		} else {
			if (eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].value") == "") {
				alert("Please enter your "+fieldnames[i]);
				eval("document.phplistSubscribeForm.elements['"+fieldstocheck[i]+"'].focus()");

				return false;
			}
		}
	}

	for (i=0;i<groupstocheck.length;i++) {
		if (!checkGroup(groupstocheck[i],groupnames[i])) {
			return false;
		}
	}
  
	if (! checkEmail()) {
		alert("Email address is not valid");
		return false;
	}

	return true;
}

var fieldstocheck = new Array();
var fieldnames = new Array();
function addFieldToCheck(value,name) {
	fieldstocheck[fieldstocheck.length] = value;
	fieldnames[fieldnames.length] = name;
}

var groupstocheck = new Array();
var groupnames = new Array();
function addGroupToCheck(value,name) {
	groupstocheck[groupstocheck.length] = value;
	groupnames[groupnames.length] = name;
}

function compareEmail() {
	return (document.phplistSubscribeForm.elements["email"].value == document.phplistSubscribeForm.elements["emailconfirm"].value);
}

function checkEmail() {
	var re = /^(([^<>()[\]\.,;:\s@\"]+(\.[^<>()[\]\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
	return re.test(document.phplistSubscribeForm.elements["email"].value);
}

function checkGroup(name,value) {
	option = -1;
	for (i=0;i<document.phplistSubscribeForm.elements[name].length;i++) {
		if (document.phplistSubscribeForm.elements[name][i].checked) {
			option = i;
		}
	}

	if (option == -1) {
		alert ("Please enter your "+value);
		return false;
	}

	return true;
}

function submitForm() {

	// first, clear the response div from the previous attempts results
	jQuery("#phplistResult").empty();

	successMessage = 'Thank you for your registration. Please check your email to confirm.';
	data = jQuery('#phplistSubscribeForm').serialize();
	jQuery.ajax( {

		type: 'POST',
		data: data,
		dataType: 'html',

		/* set the id in the url below (&id=X) to the Subscribe List ID, per
		     the "Config" -> "Subscribe pages" page in the phplist wui
		      * https://phplist.opensourceecology.org/lists/admin/?page=spage

		     Note that the Subscribe Page should use
		      * "Don't display email address confirmation field" or it will fail
		      * "Select the lists to offer" tab should check exactly one list

		     For more information, see
		      * https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
		*/
		url: 'https://phplist.opensourceecology.org/lists/index.php?p=asubscribe&id=3',

		// defines a function that's called when our ajax call suceeds (ie: gets a
		// 200 response back)
		success: function (data, status, request) {

			jQuery("#phplistResult").empty().append(data != '' ? data : successMessage);
			jQuery('#attribute1').val('');
			jQuery('#email').val('');

		 },

		// defines a function that's called when our ajax call fails (ie: gets a 500
		// response back)
		error: function (request, status, error) {

			jQuery("#phplistResult").empty();
			alert('Sorry, we were unable to process your subscription.');

		}

	});

}

</script>

<div id="phplistAjaxSubscribeFormWrapper" style="display:none; background-color:#e4e8eb; padding:20px">

	<h3>Sign up for the OSEmail newsletter:</h3>

	<!-- this simple form intentionally only requires the bare necessities:
	  [1] email address
	  [2] accept Privacy Policy
	-->
	<form method="post" action="" name="phplistSubscribeForm" id="phplistSubscribeForm">

		<!-- [1] email address -->
		<input type=text name=email required="required" placeholder="Email" size="30" id="email" />
		<script language="Javascript" type="text/javascript">addFieldToCheck("email","Email address");</script>
		<br/>

		<!-- [2] accept Privacy Policy -->
		<input type="checkbox" name="attribute4" value="on"  class="attributeinput" id="attribute4" />
		<label for="attribute4">I agree to the OSE <a href='https://wiki.opensourceecology.org/wiki/Open_Source_Ecology:Privacy_policy' target='_blank'>Privacy Policy</a></label>
		<script language="Javascript" type="text/javascript">addFieldToCheck("attribute4","I agree to the OSE Privacy Policy");</script>
		<br/>

		<!-- other strange hidden inputs per phplist's ajax example thread
		  * https://discuss.phplist.org/t/solved-ajax-subscribe-api/974
		-->
		<input type="hidden" name="list[3]" value="signup" />
		<input type="hidden" name="listname[3]" value="OSEmail"/>
		<input type="hidden" name="htmlemail" value="1" />
		<div style="display:none"><input type="text" name="VerificationCodeX" value="" size="20"></div>

	</form>

	<!-- do some input sanity checking with js, then ajax() submit using jquery -->
	<button class='button' onclick="if (checkform()) {submitForm();} return false;">Subscribe</button>

<div id="phplistResult"></div>
</div>

<!-- this input will fill-in with results from the phplist server after the ajax
submission via jquery -->


<script type="text/javascript">

// display the ajax subscription form iff js is enabled
document.getElementById( 'phplistAjaxSubscribeFormWrapper' ).style.display = 'block';

</script>

Troubleshooting

This section will provide hints for troubleshooting the phplist software. For general troubleshooting tips, also checkout the relevant documentation page on phplist.org's website here:

Enable error logging

By default, phplist will suppress errors from being written to log files. To fix this, find calls to error_reporting(0) and change them to error_reporting(1) in the phplist source code. For example, '/lists/admin/index.php' and '/lists/admin/init.php' [5]

Formatting

Address -

Open Source Ecology Global Headquarters Maysville, MO 64469 USA

Do not include street address at this time.

edit

Welcome Email

Welcome to OSEmail - the Open Source Ecology Newsletter

Please keep this message for later reference.

Your email address has been added to the following newsletter(s): 

*newsletter
*OSEmail

To update your details and preferences please go to http://phplist.opensourceecology.org/lists/?p=preferences&uid=d56f8157f023a2dc4217da85c43f9c71. If you do not want to receive any more messages, please go to http://phplist.opensourceecology.org/lists/?p=unsubscribe&uid=d56f8157f023a2dc4217da85c43f9c71.

Thank you

Repermissioning

The True Fans List

  • Deprecated - [1]
  • Going forward, add this data to phplist's (private) list #4 = True Fans, which now has 957 list subscribers. [2]

See Also

External Links

  1. https://www.phplist.org/manual/
  2. https://resources.phplist.com/documentation/configuration
  3. https://discuss.phplist.org/
  4. https://github.com/phpList/phplist3
  5. https://mantis.phplist.org/
  6. Hardening Guide https://tech.michaelaltfield.net/2020/02/14/phplist-hardening-security/
  7. fix phplist 500 error due to random_compat https://tech.michaelaltfield.net/2018/08/25/fix-phplist-500-error-due-to-random_compat/

References

  1. ↑ https://github.com/paragonie/random_compat/issues/99
  2. ↑ https://github.com/paragonie/random_compat/blob/master/RATIONALE.md
  3. ↑ https://www.w3.org/2011/webappsec/cors-draft/#resource-implementation
  4. ↑ https://discuss.phplist.org/t/any-plans-for-an-official-wordpress-plugin/3894/4
  5. ↑ https://discuss.phplist.org/t/troubleshooting-techniques-manual-chapter-feedback-and-discussion/234/2

Links

Retrieved from "https://wiki.opensourceecology.org/index.php?title=Phplist&oldid=300808"