OSE Server

From Open Source Ecology
Revision as of 00:37, 7 July 2017 by Maltfield (talk | contribs)
Jump to navigation Jump to search

Introduction

The OSE Server is a critical piece of the OSE Development Stack - thus making the (1) OSE Software Stack and the OSE Server Stack the 2 critical components of OSE's development infrastructure.

OSE Server Management

Working Doc - edit

Working Presentation -

edit

2016

Ordered with CentOS 7.2, and installing Webmin for server admin.

New OSE server, June 2016. Older server had 4 GB RAM compared to the 64 GB here.
Filezilla login and directory structure on Hetzner 2016. September 2016.

Assessment of Server Options

  • 6/16 setup on Hetzner 2011 is shit and needs updating - AMD Athlon 64 X2 5600+ Processor, 4 GB RAM, 2x 400 GB Harddisks, 1 Gbit/s Connection
  • Main figure of merit - RAM space - which is how many pages it can store in memory before having to use hard disks - where RAM access is instantaneous, and hard disk access is slow.

Proposed Solution

  • Upgrade hardware/plan on Hetzner
  • Document sysadmin to do sysadmin in house

Backups

We actively backup our server's data on a daily basis.

Important Files & Directories

The following files/directories are related to the daily backup process:

  1. /root/backups/backup.sh This is the script that preforms the backups
  2. /root/backups/sync/ This is where backup files are stored before they're rsync'd to the storage server. '/root/backups/sync*' is explicitly excluded from backups itself to prevent a recursive nightmare.
  3. /root/backups/sync.old/ This is where the files from the previous backup are stored; they're deleted by the backup script at the beginning of a new backup, and replaced by the files from 'sync'
  4. /root/backups/backup.settings This holds important variables for the backup script. Note that this file should be on heavy lockdown, as it contains critical credentials (passwords).
  5. /etc/cron.d/backup_to_dreamhost This file tells the cron server to execute the backup script at 07:20 UTC, which is roughly midnight in North America--a time of low traffic for the OSE Server
  6. /var/log/backups/backup.log The backup script logs to this file
  7. /root/.ssh/id_rsa The private ssh key used to rsync files to the dreamhost server. This file should be on lockdown, as it's a critical credential that allows read/write access to our dreamhost server over ssh.

What's backed-up

Here is what is being backed-up:

  1. mysqldump of all databases
  2. all files in /etc/*
  3. all files in /home/*
  4. all files in /var/log/*
  5. all files in /root/* (except the 'backups/sync*' dirs)
  6. all files in /var/www/*

Backup Server

As a nonprofit, we're eligible for "unlimited" storage account with dreamhost. Therefore, we rsync our backup files to our dreamhost server at the end of the backup script.

Note that we don't actually have unlimited storage on this server, and archives of TBs of data would surely be a violation of their policy. Therefore, we should be respectful of this free service & keep our total usage below 500G.

The following files/directories are related to the daily backup process on the backup server:

  1. /home/marcin_ose/backups/hetzner2/ This directory holds a set of dirs that are timestamped & hold the contents of the 'sync' directory from the hetzner2 server
  2. /home/marcin_ose/bin/cleanLocal.pl This script deletes files older than a specified age from a specified directory
  3. /home/marcin_ose/logs/cleanBackups.log This is the log file that cleanLocal.pl writes to
  4. /home/marcin_ose/.ssh/authorized_keys This file lists the public key as found in /root/.ssh/id_rsa.pub on the hetzner 2 server, and permits the backup script to write files to the dreamhost server over ssh (rsync).

Because we don't have root access to the dreamhost backup server, the cron responsible for deleting old backups is stored in the crontab. Execute `crontab -l` to see the cron config.

Note that the cleanLocal.pl script does *not* delete backup files that were created on the 1st of every month. These should periodically be cleared out manually, if space becomes an issue. Otherwise, cron is configured to call cleanLocal.pl to preserve backups for 3 days back, deleting files older than this.

TODO

As of 2017-06, the goal in the next few months is to migrate all services off of Hetzner 1, and terminate our Hetzner 1 plan entirely. The following is a set of tasks to reach this goal:

  1. Harden SSH
  2. Document how to add ssh users to Hetzner 2
  3. Statuscake
  4. Piwik
  5. OSSEC
  6. Harden Apache
  7. Harden Mysql
  8. iptables
  9. Let's Encrypt for OBI
  10. Organize & Harden Wordpress for OBI
  11. Qualys SSL labs validation && tweaking
  12. Disable Cloud Front
  13. Fine-tune Wiki config
  14. Begin migrating services to Hetzner 2

Links

Retrieved from "https://wiki.opensourceecology.org/index.php?title=OSE_Server&oldid=155159"