This article is about the migration from "Hetzner2" to "Hetzner3".

For more general information about the OSE Server, you probably want to see OSE Server.

Initial Provisioning

OS Install

We used hetzner's installimage tool to install Debian 12 on hetzner3.

We kept all the defaults, except the hostname.

The two NVMe disks were setup in a software RAID1 with a 32G swap, 1G '/boot', and the rest for '/'.

Initial Hardening

After the OS's first boot, I (Michael Altfield) ran a quick set of commands to create a user for me, do basic ssh hardening, and setup a basic firewall to block everything except ssh

adduser maltfield --disabled-password --gecos ''
groupadd sshaccess
gpasswd -a maltfield sshaccess
mkdir /home/maltfield/.ssh/
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGNYjR7UKiJSAG/AbP+vlCBqNfQZ2yuSXfsEDuM7cEU8PQNJyuJnS7m0VcA48JRnpUpPYYCCB0fqtIEhpP+szpMg2LByfTtbU0vDBjzQD9mEfwZ0mzJsfzh1Nxe86l/d6h6FhxAqK+eG7ljYBElDhF4l2lgcMAl9TiSba0pcqqYBRsvJgQoAjlZOIeVEvM1lyfWfrmDaFK37jdUCBWq8QeJ98qpNDX4A76f9T5Y3q5EuSFkY0fcU+zwFxM71bGGlgmo5YsMMdSsW+89fSG0652/U4sjf4NTHCpuD0UaSPB876NJ7QzeDWtOgyBC4nhPpS8pgjsnl48QZuVm6FNDqbXr9bVk5BdntpBgps+gXdSL2j0/yRRayLXzps1LCdasMCBxCzK+lJYWGalw5dNaIDHBsEZiK55iwPp0W3lU9vXFO4oKNJGFgbhNmn+KAaW82NBwlTHo/tOlj2/VQD9uaK5YLhQqAJzIq0JuWZWFLUC2FJIIG0pJBIonNabANcN+vq+YJqjd+JXNZyTZ0mzuj3OAB/Z5zS6lT9azPfnEjpcOngFs46P7S/1hRIrSWCvZ8kfECpa8W+cTMus4rpCd40d1tVKzJA/n0MGJjEs2q4cK6lC08pXxq9zAyt7PMl94PHse2uzDFhrhh7d0ManxNZE+I5/IPWOnG1PJsDlOe4Yqw== maltfield@ose" > /home/maltfield/.ssh/authorized_keys
chown -R maltfield:maltfield /home/maltfield/.ssh
chmod -R 0600 /home/maltfield/.ssh
chmod 0700 /home/maltfield/.ssh

# without this, apt-get may get stuck
export DEBIAN_FRONTEND=noninteractive

apt-get update
apt-get -y install iptables iptables-persistent
apt-get -y purge nftables

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j DROP
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 32415 -j ACCEPT
iptables -A INPUT -j DROP

iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j ACCEPT
iptables -A OUTPUT -m owner --uid-owner 0 -j ACCEPT
iptables -A OUTPUT -m owner --uid-owner 42 -j ACCEPT
iptables -A OUTPUT -m owner --uid-owner 1000 -j ACCEPT
iptables -A OUTPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
iptables -A OUTPUT -j DROP

ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -s ::1/128 -d ::1/128 -j DROP
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -j DROP

ip6tables -A OUTPUT -s ::1/128 -d ::1/128 -j ACCEPT
ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A OUTPUT -m owner --uid-owner 0 -j ACCEPT
ip6tables -A OUTPUT -m owner --uid-owner 42 -j ACCEPT
ip6tables -A OUTPUT -m owner --uid-owner 1000 -j ACCEPT
ip6tables -A OUTPUT -j DROP

iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig.`date "+%Y%m%d_%H%M%S"`
grep 'Port 32415' /etc/ssh/sshd_config || echo 'Port 32415' >> /etc/ssh/sshd_config
grep 'AllowGroups sshaccess' /etc/ssh/sshd_config || echo 'AllowGroups sshaccess' >> /etc/ssh/sshd_config
grep 'PermitRootLogin no' /etc/ssh/sshd_config || echo 'PermitRootLogin no' >> /etc/ssh/sshd_config
grep 'PasswordAuthentication no' /etc/ssh/sshd_config || echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config
systemctl restart sshd.service

apt-get -y upgrade

After all the packages updated, I gave my new user sudo permission

root@mail ~ # cp /etc/sudoers /etc/sudoers.20240731.orig
root@mail ~ # 

root@mail ~ # visudo
root@mail ~ # 

root@mail ~ # diff /etc/sudoers.20240731.orig /etc/sudoers
47a48
> maltfield ALL=(ALL:ALL) NOPASSWD:ALL
root@mail ~ #

Ansible

TODO

Purchase

We purchased Hetzner3 from a Dedicated Server Auction on 2024-07-30 for 37.72 EUR/mo.

Hardware

Hetzner3 came with the following hardware:

* Intel Core i7-6700
* 2x SSD M.2 NVMe 512 GB
* 4x RAM 16384 MB DDR4
* NIC 1 Gbit Intel I219-LM
* Location: Germany
* Rescue system (English)
* 1 x Primary IPv4

CPU

Hetzner3 has a Intel Core i7-6700. It's a 4-core + 8-thread 3.4 Ghz processor from 2015 with 8M Cache. This cannot be upgraded.

TODO: cat /cpu/proc

For comparison, this is the same processor that we've been using in Hetzner2, and it's way over-provisioned for our needs.

Disk

2x 512 GB NVMe disks should suit us fine.

We also have one empty NVMe slot and two emtpy SATA slots. As of today, we can upgrade each SATA slot with a max 3.84 TB SSD or max 22 TB HDD.

For comparison, we had 2x 250 GB SSD disks in Hetzner2, so this should be approximately double the capacity and a somewhat better disk io ops.

Memory

We have 64 GB of DDR4 RAM.

TODO: figure out if this is upgradeable

This is the same

For comparison, this is the same memory as we've been using in Hetzner2. We could get-by with less, but varnish is happy to use it.

Initial Specifications Research

Because hetzner2 ran on CentOS7 (which was EOL'd 2024-06-30), Marcin asked Michael in July 2024 to begin provisioning a "hetzner3" with Debian to replace "hetzner2".

Munin

I (Michael Altfield) collected some charts from munin to confirm my understanding of the server's resource needs before purchasing a new dedicated server from Hetzner.

CPU

In 2018^[1], I said we'd want min 2-4 cores.

After reviewing the cpu & load charts for the past year, load rarely ever touches 3. Most of the time it hovers between 0.2 - 1. So I agree that 4 cores is fine for us now.

Most of these auctions have a Intel Core i7-4770, which is a 4-core + 8 thread proc. That should be fine.

Disk

Honestly, I expect that the lowest offerings of a dedicated server in 2024 are probably going to suffice for us, but what I'm mostly concerned-about is the disk. Even last week when I did the yum updates, I nearly filled the disk just by extracting a copy of our backups. Currently we have two 250G disks in a software RAID-1 (mirror) array. That give us a useable 197G

It's important to me that we double this at-least, but I'll see if there's any deals on 1TB disks or larger.

Also what we currently have is a 6 Gb/s SSD, so I don't want to downgrade that by going to a spinning-disk HDD. NvME might be a welcome upgrade. I/O wait is probably a bottleneck, but not currently one that's causing us agony

To be clear: the usage line of '/' in this chart is the middle-green line, which is ~50% full

Memory

In 2018^[1], I said we'd want 8-16G RAM minimum. While that's technically true, we currently have 64G RAM. Most of these base cheap-as-they-come dedicated servers in the hetzener auction page have 64G RAM.

We use 40G of RAM just for varnish, which [a] greatly reduces load on the server and [b] gives our read-only visitors a much, much faster page load time. While we don't strictly *need* that much RAM, I'm going to make sure hetzner3 has at least as much RAM as hetzner2.