CHG-2025-04-30 replace hetzner2 sda - Revision history

Maltfield: /* Status */

2025-04-30T14:29:40Z

Status

← Older revision		Revision as of 14:29, 30 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-30 14:30 UTC==

			This change was completed successfully

	==2025-04-30 14:18 UTC==		==2025-04-30 14:18 UTC==

Maltfield: /* Status */

2025-04-30T14:29:17Z

Status

← Older revision		Revision as of 14:29, 30 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-30 14:18 UTC==

			# I'm going to double-tap the grub install before giving it a reboot
			<pre>
			[root@opensourceecology ~]# grub2-install /dev/sda
			Installing for i386-pc platform.
			grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
			grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
			Installation finished. No error reported.
			[root@opensourceecology ~]#
			</pre>
			# and I rebooted it
			<pre>
			[root@opensourceecology ~]# reboot
			Connection to opensourceecology.org closed by remote host.
			Connection to opensourceecology.org closed.
			ssh: connect to host opensourceecology.org port 32415: Connection refused
			ssh: connect to host opensourceecology.org port 32415: Connection refused
			...
			user@personal:~$ autossh opensourceecology.org
			Last login: Wed Apr 30 11:28:26 2025 from REDACTED
			[maltfield@opensourceecology ~]$ uptime
			14:17:14 up 1 min, 1 user, load average: 0.85, 0.24, 0.08
			[maltfield@opensourceecology ~]$
			</pre>
			# cool, it came back.
			# cool, raid looks healthy
			<pre>
			[root@opensourceecology ~]# cat /proc/mdstat
			Personalities : [raid1]
			md2 : active raid1 sda3[3] sdb3[2]
			209984640 blocks super 1.2 [2/2] [UU]
			bitmap: 2/2 pages [8KB], 65536KB chunk

			md0 : active raid1 sdb1[2] sda1[3]
			33521664 blocks super 1.2 [2/2] [UU]

			md1 : active raid1 sdb2[2] sda2[3]
			523712 blocks super 1.2 [2/2] [UU]

			unused devices: <none>
			[root@opensourceecology ~]#
			[root@opensourceecology ~]# lsblk
			NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
			sda 8:0 0 477G 0 disk
			├─sda1 8:1 0 32G 0 part
			│ └─md0 9:0 0 32G 0 raid1 [SWAP]
			├─sda2 8:2 0 512M 0 part
			│ └─md1 9:1 0 511.4M 0 raid1 /boot
			└─sda3 8:3 0 200.4G 0 part
			└─md2 9:2 0 200.3G 0 raid1 /
			sdb 8:16 0 477G 0 disk
			├─sdb1 8:17 0 32G 0 part
			│ └─md0 9:0 0 32G 0 raid1 [SWAP]
			├─sdb2 8:18 0 512M 0 part
			│ └─md1 9:1 0 511.4M 0 raid1 /boot
			└─sdb3 8:19 0 200.4G 0 part
			└─md2 9:2 0 200.3G 0 raid1 /
			[root@opensourceecology ~]#
			</pre>
			# and SMART isn't yelling about failed disks anymore
			<pre>
			[root@opensourceecology ~]# smartctl -H /dev/sda
			smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1160.119.1.el7.x86_64] (local build)
			Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

			=== START OF READ SMART DATA SECTION ===
			SMART overall-health self-assessment test result: PASSED

			[root@opensourceecology ~]# smartctl -H /dev/sdb
			smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1160.119.1.el7.x86_64] (local build)
			Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

			=== START OF READ SMART DATA SECTION ===
			SMART overall-health self-assessment test result: PASSED

			[root@opensourceecology ~]#
			</pre>

	==2025-04-30 14:13 UTC==		==2025-04-30 14:13 UTC==

Maltfield: /* Status */

2025-04-30T14:27:33Z

Status

← Older revision		Revision as of 14:27, 30 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-30 14:13 UTC==

			The RAID sync is finished; I guess these Micron 500G disks have better i/o throughput than our old 200GCrucial disks

			<pre>
			Wed Apr 30 14:07:12 UTC 2025
			Personalities : [raid1]
			md0 : active raid1 sda1[3] sdb1[2]
			33521664 blocks super 1.2 [2/1] [_U]
			[====>................] recovery = 21.2% (7124992/33521664) finish=2.2min speed=191533K/sec

			md2 : active raid1 sda3[3] sdb3[2]
			209984640 blocks super 1.2 [2/2] [UU]
			bitmap: 1/2 pages [4KB], 65536KB chunk

			md1 : active raid1 sda2[3] sdb2[2]
			523712 blocks super 1.2 [2/2] [UU]

			unused devices: <none>

			Wed Apr 30 14:12:12 UTC 2025
			Personalities : [raid1]
			md0 : active raid1 sda1[3] sdb1[2]
			33521664 blocks super 1.2 [2/2] [UU]

			md2 : active raid1 sda3[3] sdb3[2]
			209984640 blocks super 1.2 [2/2] [UU]
			bitmap: 2/2 pages [8KB], 65536KB chunk

			md1 : active raid1 sda2[3] sdb2[2]
			523712 blocks super 1.2 [2/2] [UU]

			unused devices: <none>
			</pre>

	==2025-04-30 13:48 UTC==		==2025-04-30 13:48 UTC==

Maltfield: /* Status */

2025-04-30T14:24:22Z

Status

Show changes

Maltfield: /* Status */

2025-04-30T14:22:41Z

Status

← Older revision		Revision as of 14:22, 30 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-30 13:26 UTC==

			# I got a response back from hetzner 4 minutes later
			<pre>
			Dear Client.

			We do not have these drives "new" anymore. Therefore, this is not possible. We already selected a drive with less than 20.000h. We also did not charge the fee for a new drive.
			</pre>
			# so it looks like we got the drive free, but that's still nearly a waste of my time. I replied and asked them how long it would take for them to order a new drive
			<pre>
			I emailed last week about this to make sure you had time to order a new drive (check my support tickets).

			This drive you inserted has only 32% of its life left, according to SMART. It's closer to dead than new.

			How long would it take you to order a new drive?
			</pre>


	==2025-04-30 13:20 UTC==		==2025-04-30 13:20 UTC==

Maltfield: hetzner gave us a very used drive :(

2025-04-30T14:21:15Z

hetzner gave us a very used drive :(

← Older revision		Revision as of 14:21, 30 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-30 13:20 UTC==

			We're still waiting on hetzner.

			Hetzner replaced the drive with one that already has been used for 18,623 hours, which means it has only 32% of its life left.

			<pre>
			[root@opensourceecology ~]# smartctl -A /dev/sda
			smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1160.119.1.el7.x86_64] (local build)
			Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

			=== START OF READ SMART DATA SECTION ===
			SMART Attributes Data Structure revision number: 16
			Vendor Specific SMART Attributes with Thresholds:
			ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
			1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
			5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
			9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 18623
			12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 9
			171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
			172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
			173 Ave_Block-Erase_Count 0x0032 032 032 000 Old_age Always - 1030
			174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 2
			183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
			184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
			187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
			194 Temperature_Celsius 0x0022 068 047 000 Old_age Always - 32 (Min/Max 23/53)
			196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
			197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
			198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
			199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0
			202 Percent_Lifetime_Remain 0x0030 032 032 001 Old_age Offline - 68
			206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
			246 Total_Host_Sector_Write 0x0032 100 100 000 Old_age Always - 96994281182
			247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 3059820027
			248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 31429771271
			180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 2467
			210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0

			[root@opensourceecology ~]# date -u
			Wed Apr 30 13:23:39 UTC 2025
			[root@opensourceecology ~]#
			</pre>

			In the remote-hands support request, I was very clear that they should replace it with a new drive (with <1,000 hours of use). We're paying for that.

			I asked them to insert an actually new drive with <1,000 hours of use.

	==2025-04-30 11:44 UTC==		==2025-04-30 11:44 UTC==

Maltfield: status update: removed sda from RAID and submitted remote hands request to swap disk

2025-04-30T11:45:02Z

status update: removed sda from RAID and submitted remote hands request to swap disk

Show changes

Maltfield: change begun

2025-04-30T11:30:36Z

change begun

← Older revision		Revision as of 11:30, 30 April 2025
Line 1:		Line 1:
	=Status=		=Status=

			==2025-04-30 11:29 UTC==

			Starting Change

	==2025-04-24 17:56 UTC==		==2025-04-24 17:56 UTC==

Maltfield: updated CHG start time to 2025-04-30 11:00 UTC

2025-04-24T17:49:36Z

updated CHG start time to 2025-04-30 11:00 UTC

@@ Line 1: / Line 1: @@
 =Status=
 ==2025-04-24 17:37 UTC==
@@ Line 51: / Line 93: @@
 ==Scheduled Time==
-This change will take place on 2025-??-?? ??:00 UTC
+This change will take place on 2025-04-30 11:00 UTC
-* = 2025-??-?? ??:00 Kansas City, US
+* = 2025-04-30 06:00 Kansas City, US
-* = 2025-??-?? ??:00 Guayaquil, EC
+* = 2025-04-30 06:00 Guayaquil, EC
-https://www.timeanddate.com/worldclock/converter.html?iso=20240727T160000&p1=405&p2=1440&p3=93
+https://www.timeanddate.com/worldclock/converter.html?iso=20250430T110000&p1=405&p2=1440&p3=93
 ==Purpose==

Maltfield: Maltfield moved page CHG-2025-XX-XX replace hetzner2 sda to CHG-2025-04-30 replace hetzner2 sda: marcin approved the date & time of this CHG

2025-04-24T17:46:58Z

Maltfield moved page CHG-2025-XX-XX replace hetzner2 sda to CHG-2025-04-30 replace hetzner2 sda: marcin approved the date & time of this CHG

← Older revision	Revision as of 17:46, 24 April 2025
(No difference)