CHG-2025-04-24 replace hetzner2 sdb - Revision history

Maltfield: /* See Also */

2025-04-24T17:47:28Z

2025-04-24T16:36:16Z

2025-04-24 16:18 UTC

@@ Line 31: / Line 31: @@
 md1 : active raid1 sdb2[2] sda2[0]
 blocks super 1.2 [2/2] [UU]
 unused devices: <none>
 </pre>

2025-04-24T16:26:49Z

completed successfully

← Older revision		Revision as of 16:26, 24 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-24 16:18 UTC==

			The new disk is now fully in-sync with the old (failing) disk, since sometime between 15:15 and 15:20 UTC

			<pre>
			Thu Apr 24 15:15:59 UTC 2025
			Personalities : [raid1]
			md2 : active raid1 sdb3[2] sda3[0]
			209984640 blocks super 1.2 [2/1] [U_]
			[===================>.] recovery = 96.5% (202794752/209984640) finish=2.5min speed=46324K/sec
			bitmap: 2/2 pages [8KB], 65536KB chunk

			md0 : active raid1 sdb1[2] sda1[0]
			33521664 blocks super 1.2 [2/2] [UU]

			md1 : active raid1 sdb2[2] sda2[0]
			523712 blocks super 1.2 [2/2] [UU]

			unused devices: <none>

			Thu Apr 24 15:20:59 UTC 2025
			Personalities : [raid1]
			md2 : active raid1 sdb3[2] sda3[0]
			209984640 blocks super 1.2 [2/2] [UU]
			bitmap: 1/2 pages [4KB], 65536KB chunk

			md0 : active raid1 sdb1[2] sda1[0]
			33521664 blocks super 1.2 [2/2] [UU]

			md1 : active raid1 sdb2[2] sda2[0]
			523712 blocks super 1.2 [2/2] [UU]

			unused devices: <none>
			</pre>

			I'm marking this change as completed successfully. Next-up we replace the other failing disk. See [[CHG-2025-XX-XX_replace_hetzner2_sda]]

	==2025-04-24 14:23 UTC==		==2025-04-24 14:23 UTC==

2025-04-24T14:30:43Z

2025-04-24 14:23 UTC

← Older revision		Revision as of 14:30, 24 April 2025
Line 29:		Line 29:

	I was able to partition the new disk and add it to a RAID. At the time of writing, both swap and boot are synced (and grub installed on the new disk), and it's still syncing the root partition on the new disk in the RAID (currently at 35% and writing at 58 MB/s		I was able to partition the new disk and add it to a RAID. At the time of writing, both swap and boot are synced (and grub installed on the new disk), and it's still syncing the root partition on the new disk in the RAID (currently at 35% and writing at 58 MB/s


	When the backup finished uploading, I put the web services back online and typed this status message.		When the backup finished uploading, I put the web services back online and typed this status message.

2025-04-24T14:30:31Z

Status

← Older revision		Revision as of 14:30, 24 April 2025
Line 29:		Line 29:

	I was able to partition the new disk and add it to a RAID. At the time of writing, both swap and boot are synced (and grub installed on the new disk), and it's still syncing the root partition on the new disk in the RAID (currently at 35% and writing at 58 MB/s		I was able to partition the new disk and add it to a RAID. At the time of writing, both swap and boot are synced (and grub installed on the new disk), and it's still syncing the root partition on the new disk in the RAID (currently at 35% and writing at 58 MB/s


			When the backup finished uploading, I put the web services back online and typed this status message.

	==2025-04-24 10:32 UTC==		==2025-04-24 10:32 UTC==

2025-04-24T14:29:02Z

Status

← Older revision		Revision as of 14:29, 24 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-24 14:23 UTC==

			The wiki is back!

			Unfortunately, hetzner fucked-up and removed both disks

			<pre>
			Dear Client,

			we've replaced the drive via hotswap as wished.

			The second drive was unfortunately also briefly disconnected as there was a=
			wrong physical label on it.

			If you have any further questions or problems, feel free to contact us agai=
			n.


			Kind regards

			Nils Wei=C3=9Fer
			</pre>

			The result was that /dev/sda was listed as /dev/sdc, the new drive was /dev/sdb, and dmesg was being spammed with I/O and RAID errors. The wiki was down. Disks were read-only, so I couldn't even take backups. I tried to reboot, but even <code>reboot</code> failed due to i/o errors.

			I used the WUI to trigger a reboot, and--thank god--the server came-up again. I immediately took down all the web services as I investigated the damage and triggered a new backup.

			I was able to partition the new disk and add it to a RAID. At the time of writing, both swap and boot are synced (and grub installed on the new disk), and it's still syncing the root partition on the new disk in the RAID (currently at 35% and writing at 58 MB/s

	==2025-04-24 10:32 UTC==		==2025-04-24 10:32 UTC==

2025-04-24T10:33:42Z

status update 10:32 UTC

← Older revision		Revision as of 10:33, 24 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-24 10:32 UTC==

			I finished submitting the request to hetnzer to replace the disk for free.

			It says we should expect the new disk to be inserted in 2-4 hours. One part of the form said this would happen without downtime. But the (required) checkbox at the bottom said that I understand that downtime is required. So that's ambiguous.

	==2025-04-24 10:22 UTC==		==2025-04-24 10:22 UTC==

2025-04-24T10:23:00Z

status update 10:22 UTC

@@ Line 1: / Line 1: @@
 =Status=
 ==2025-04-24 10:07 UTC==
@@ Line 196: / Line 260: @@
 mdadm /dev/md0 -r /dev/sdb1
 mdadm /dev/md1 -r /dev/sdb2
-mdadm /dev/md1 -r /dev/sdb3
+mdadm /dev/md2 -r /dev/sdb3
 </pre>

2025-04-24T10:16:56Z

fix typo in removeal commands :(

2025-04-24T10:07:54Z

← Older revision		Revision as of 10:07, 24 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-24 10:07 UTC==

			I confirmed that the RAID looks healthy, and our daily backups finished a few hours ago

			<pre>
			[root@opensourceecology ~]# cat /proc/mdstat
			Personalities : [raid1]
			md1 : active raid1 sda2[0] sdb2[1]
			523712 blocks super 1.2 [2/2] [UU]

			md0 : active raid1 sda1[0] sdb1[1]
			33521664 blocks super 1.2 [2/2] [UU]

			md2 : active raid1 sda3[0] sdb3[1]
			209984640 blocks super 1.2 [2/2] [UU]
			bitmap: 2/2 pages [8KB], 65536KB chunk

			unused devices: <none>
			[root@opensourceecology ~]#

			[root@opensourceecology ~]# source /root/backups/backup.settings
			[root@opensourceecology ~]# ${RCLONE} ls "b2:${B2_BUCKET_NAME}" \| grep $(date "+%Y%m%d")
			20144027578 daily_hetzner3_20250424_074924.tar.gpg
			[root@opensourceecology ~]#

			[root@opensourceecology ~]# date -u
			Thu Apr 24 10:06:52 UTC 2025
			[root@opensourceecology ~]#
			</pre>

	==2025-04-24 10:04 UTC==		==2025-04-24 10:04 UTC==

← Older revision		Revision as of 17:47, 24 April 2025
Line 548:		Line 548:

	# [[Maltfield_Log/2025_Q2]] Investigation into failed disks (after db corruption event in April)		# [[Maltfield_Log/2025_Q2]] Investigation into failed disks (after db corruption event in April)
			# [[CHG-2025-04-30_replace_hetzner2_sda]] replacement of the second disk
	# [[:Category: CHGs\|List of other CHG "tickets"]]		# [[:Category: CHGs\|List of other CHG "tickets"]]