OSE Server

Introduction

The OSE Server is a critical piece of the OSE Development Stack - thus making the (1) OSE Software Stack and the OSE Server Stack the 2 critical components of OSE's development infrastructure.

Uptime & Status Checks

If you think one of the OSE websites or services may be offline, you can verify their status at the following site:

* http://status.opensourceecology.org/

Note that this URL is just a convenient CNAME to uptime.statuscake.com, which is configured to redirect our CNAME to our Public Reporting Dashboard here:

* https://uptime.statuscake.com/?TestID=itmHX7Pfj2

It may be a good idea to bookmark the above URL in the event that our site goes down, possibly including DNS issues preventing the CNAME redirect from status.opensourceecolgy.org

Note that Statuscake also tracks uptime over months, and can send monthly uptime reports, as well as immediate email alerts when the site(s) go down. If you'd like to receive such alerts, contact the OSE System Administrator.

Adding Statuscake Checks

To modify our statuscake checks, you should login to the statuscake website using the credentials stored in keepass.

If you want the test to be public (appearing on http://status.opensourceecology.org), you should add it by editing the Public Reporting Dashboard.

OSE Server Management

Working Doc - edit

Working Presentation -

edit

2016

Ordered with CentOS 7.2, and installing Webmin for server admin.

New OSE server, June 2016. Older server had 4 GB RAM compared to the 64 GB here.

Filezilla login and directory structure on Hetzner 2016. September 2016.

Assessment of Server Options

• 6/16 setup on Hetzner 2011 is shit and needs updating - AMD Athlon 64 X2 5600+ Processor, 4 GB RAM, 2x 400 GB Harddisks, 1 Gbit/s Connection

• Main figure of merit - RAM space - which is how many pages it can store in memory before having to use hard disks - where RAM access is instantaneous, and hard disk access is slow.

• Difference Between a Dedicated and Managed Server
• HDD vs SSD Storage
• DDOS Attacks are an issue according to Y Combinator News - [1]
• Latency on Hetzner in different parts of the world - [2]

Proposed Solution

• Upgrade hardware/plan on Hetzner
• Document sysadmin to do sysadmin in house

SSH

Our server has ssh access. If you require access to ssh, contact the OSE System Administrator with subject "ssh access request," and include the following information in the body of the email:

1. An explanation as to why you need ssh access
2. What you need access to
3. Provide a link to a portfolio of prior experience working with linux over command line that demonstrates your experience & competency using the command line safely
4. Provide a few references for previous work in which you had experience working with linux over command line

Add new users

The following steps will add a new user to the OSE Server.

First, create the new user. Generate & set a temporary, 100-character, random, alpha-numeric password for the user.

useradd <new_username>
passwd <new_username>

Only if it's necessary, send this password to the user through a confidential/encrypted medium (ie: the Wire app). They would need it if they want to reset their password. Note that they will not be able to authenticate with their password over ssh, and this is intentional. In fact, it is unlikely they will need their password at all, unless perhaps they will require sudo access. For this reason, it's best to set this password "just in case," not save it, and not send it to the user--it's more likely to confuse them. If they need their password for some reason in the future, you can reset it to a new random password in the future as the root user, and send it to them over an encrypted medium.

If the user needs ssh access, add them to the 'sshaccess' group.

gpasswd -a <new_username> sshaccess

Have the user generate a strong rsa keypair using the following command. Make sure they have it encrypted with a strong passphrase--to ensure they have 2FA. Then have them send you their new public key. The following commands should be run on the new user's computer, not the server:

ssh-keygen -t rsa -b 4096 -o -a 100
cat /home/<username>/.ssh/id_rsa.pub

The output from the `cat` command above is their public key. Have them send this to you. They can use an insecure medium such as email, as there is no reason to keep the public key confidential. They should never, ever send their private key (/home/<username>/.ssh/id_rsa) to anyone. Moreover, the private key should not be copied to any other computer, except in an encrypted backup. Note this means that the user should not copy their private key to OSE servers--that's what ssh agents are for.

Now, add the ssh public key provided by the user to their authorized_keys file on the OSE Server, and set the permissions:

cd /home/<new_username>
mkdir /home/<new_username>/.ssh
vim /home/<new_username>/.ssh/authorized_keys
chown -R <new_username>:<new_username> /home/<new_username>/.ssh
chmod 700 /home/<new_username>/.ssh
chmod 644 /home/<new_username>/.ssh/authorized_keys

If the user needs sudo permissions, edit the sudoers file. This should only be done in very, very, very rare cases for users who have >5 years of experience working as a Linux Systems Administrator. Users with sudo access must be able to demonstrate a very high level of trust, experience, and competence working on the command line in a linux environment.

Backups

We actively backup our server's data on a daily basis.

Important Files & Directories

The following files/directories are related to the daily backup process:

1. /root/backups/backup.sh This is the script that preforms the backups
2. /root/backups/sync/ This is where backup files are stored before they're rsync'd to the storage server. '/root/backups/sync*' is explicitly excluded from backups itself to prevent a recursive nightmare.
3. /root/backups/sync.old/ This is where the files from the previous backup are stored; they're deleted by the backup script at the beginning of a new backup, and replaced by the files from 'sync'
4. /root/backups/backup.settings This holds important variables for the backup script. Note that this file should be on heavy lockdown, as it contains critical credentials (passwords).
5. /etc/cron.d/backup_to_dreamhost This file tells the cron server to execute the backup script at 07:20 UTC, which is roughly midnight in North America--a time of low traffic for the OSE Server
6. /var/log/backups/backup.log The backup script logs to this file
7. /root/.ssh/id_rsa The private ssh key used to rsync files to the dreamhost server. This file should be on lockdown, as it's a critical credential that allows read/write access to our dreamhost server over ssh.

What's backed-up

Here is what is being backed-up:

1. mysqldump of all databases
2. all files in /etc/*
3. all files in /home/*
4. all files in /var/log/*
5. all files in /root/* (except the 'backups/sync*' dirs)
6. all files in /var/www/*

Backup Server

As a nonprofit, we're eligible for "unlimited" storage account with dreamhost. Therefore, we rsync our backup files to our dreamhost server at the end of the backup script.

Note that we don't actually have unlimited storage on this server, and archives of TBs of data would surely be a violation of their policy. Therefore, we should be respectful of this free service & keep our total usage below 500G.

The following files/directories are related to the daily backup process on the backup server:

1. /home/marcin_ose/backups/hetzner2/ This directory holds a set of dirs that are timestamped & hold the contents of the 'sync' directory from the hetzner2 server
2. /home/marcin_ose/bin/cleanLocal.pl This script deletes files older than a specified age from a specified directory
3. /home/marcin_ose/logs/cleanBackups.log This is the log file that cleanLocal.pl writes to
4. /home/marcin_ose/.ssh/authorized_keys This file lists the public key as found in /root/.ssh/id_rsa.pub on the hetzner 2 server, and permits the backup script to write files to the dreamhost server over ssh (rsync).

Because we don't have root access to the dreamhost backup server, the cron responsible for deleting old backups is stored in the crontab. Execute `crontab -l` to see the cron config.

Note that the cleanLocal.pl script does *not* delete backup files that were created on the 1st of every month. These should periodically be cleared out manually, if space becomes an issue. Otherwise, cron is configured to call cleanLocal.pl to preserve backups for 3 days back, deleting files older than this.

Keepass

Whenever possible, we should utilize per-user credentials for logins so there is a user-specific audit trail and we have user-specific authorization-revocation abilities. However, where this is not possible, we should store usernames & passwords that our OSE Server infrastructure depends on in a secure & shared location. At OSE, we store such passwords in an encrypted keepass database that lives on the server.

passwords.kdbx file

The passwords.kdbx file is encrypted; if an attacker obtains this file, they will not be able to access any useful information. That said, we keep it in a central location on the OSE Server behind lock & key for a few reasons:

1. The OSE Server already has nightly backups, so keeping the passwords.kdbx on the server simplifies maintenance by reusing existing backup procedures for the keepass file
2. By keeping the file in a central location & updating it with sshfs, we can prevent forks & merges of per-person keepass files, which would complicate maintenance
3. The keepass file is available on a need-to-have basis to those with ssh authorization that have been added to the 'keepass' group.

The passwords.kdbx file should be owned by the user 'root' and the group 'keepass'. It should have the file permissions of 660 (such that it can be read & written by 'root' and users in the 'keepass' group, but not accessible in any way from anyone else).

The passwords.kdbx file should exist in a directory '/etc/keepass', which is owned by the user 'root' and the group 'keepass'. This directory should have permissions 770 (such that it can be read, written, & executed by 'root' and users in the 'keepass' group, but not accessible in any way from anyone else).

TODO

As of 2017-06, the goal in the next few months is to migrate all services off of Hetzner 1, and terminate our Hetzner 1 plan entirely. The following is a set of tasks to reach this goal:

1. Backups
2. Harden SSH
3. Document how to add ssh users to Hetzner 2
4. Statuscake
5. Piwik
6. OSSEC
7. Harden Apache
8. Harden PHP
9. Harden Mysql
10. iptables
11. Let's Encrypt for OBI
12. Organize & Harden Wordpress for OBI
13. Qualys SSL labs validation && tweaking
14. Disable Cloud Front
15. Fine-tune Wiki config
16. Keepass solution + documentation
17. Migrate forum to hetzner2
18. Migrate oswh to hetzner2
19. Migrate fef to hetzner2
20. Migrate wiki to hetzner2
21. Migrate osemain to hetzner2
22. Block hetzner1 traffic to all services (though easily revertible)
23. Harden forum
24. Harden oswh
25. Harden fef
26. Harden osemain
27. Harden wiki
28. End Hetzner1 contract
29. Install Jitsi Videobridge

Links

• OSE Website
• GVCS Server Requirements
• Internet, Phone, Domains, Server
• Dedicated Server Options