Removing, replacing, and resyncing a disk in a degraded RAID array 

This guide will explain how to remove the faulty disk from the existing RAID array, add a new disk, and synchronize it.

Our data center constantly monitors RAID arrays for the operating systems we provide. If we detect a RAID array degradation or failure event, we will promptly notify the customer and recommend scheduling maintenance to replace the failed disk. The replacement process usually takes around 15-20 minutes and should not lead to data loss. As long as the remaining disks continue functioning, the new disk will copy any data, allowing the RAID array to continue functioning without data loss.

Some key points to consider: 

  • This guide applies to various RAID configurations, including RAID 1, RAID 5, and RAID 10. It covers the process for a software-type RAID array. 
  • RAID arrays provide data redundancy and improved performance, but a degraded array can compromise data integrity. Properly removing, replacing, and resynchronizing hard drives is critical to maintaining the health of your RAID system. 
  • Ensure you have “smartmontools” installed, as it is required to retrieve the serial number of the failed drive.   

Step 1: Check the current RAID array status 

  1. Open your terminal. 
  1. Connect to your server. 
  1. Check the current RAID array status (see Fig. 1)

$cat /proc/mdstat Fig. 1. Checking the current RAID array status inside the terminal. 

This command will display the status of your RAID arrays. The faulty drive could be marked as [F] or might have disappeared from the RAID array. 

 

Step 2: Mark the faulty disk as failed 

a) Identify the faulty disk. 

Use the $cat /proc/mdstat command to check for the disk marked as [F]

If a drive is missing, use the $lsblk command to identify the missing disk. 

b) Mark the disk as failed (see Fig. 2)

$mdadm --manage /dev/md0 --fail /dev/sdb2 

Fig. 2. Marking /dev/sdb2 as a faulty disk. 

Replace /dev/md0 with your RAID array device and /dev/sdb2 with the faulty disk.

 

Step 3: Install “smartmontools” and retrieve the serial number of the faulty disk (if not already installed) 

  • Install “smartmontools”: 

For Debian/Ubuntu operating systems: 

$sudo apt-get install smartmontools 

For CentOS/RHEL operating systems: 

$sudo yum install smartmontools 

For Fedora

$sudo dnf install smartmontools 

For Arch Linux

$sudo pacman -S smartmontools 

  • Find the serial number of the faulty disk (see Fig. 3)

$smartctl -a /dev/sdb2 | grep "Serial Number" 

Fig. 3. Finding the serial number of /dev/sdb2 using ”smartmontool”. 

Replace /dev/sdb2 with the faulty disk's device name. This command will display the faulty disk's serial number. 

 

Step 4: Remove the faulty disk from the array 

  • Remove the disk from the array
(see Fig. 4)

$mdadm --manage /dev/md0 --remove /dev/sdb2 

Fig. 4. Removing the faulty disk from the array. 

This command removes the failed disk from the RAID array. 

  • Verify the removal

(see Fig. 5)

$cat /proc/mdstat 

Fig. 5. Verifying that the disk has been removed from the array. 

Ensure the faulty disk is no longer part of the RAID array. 

 

Step 5: Contact support and power off the server 

  • Contact technical support: 

Inform them that you have completed all the above steps and the server is ready for maintenance. 

  • Power off the server: 

$sudo shutdown -h now 

Technical support will replace the faulty drive with a new one and power the server back on. 

 

Step 6: Add the new disk 

  • Power on the server if it is not already powered on. 
  • Copy the partition table from a good disk to the new disk. Identify the good disk (e.g.,
/dev/sda) and the new disk (e.g., /dev/sdb): 

$ sfdisk -d /dev/sda | sfdisk /dev/sdb 

Replace /dev/sda with the good disk's device name and /dev/sdb with the new disk's device name. 

  • Verify the partition table on the new disk matches the other disks in the array
(see Fig. 6): 

$fdisk –l 

Fig. 6. Checking the file system of the new drive to ensure it matches other disks. 

Confirm the filesystem of the new disk is the same as the other disks in the array. 

  • Add the new disk to the RAID array
(see Fig. 7)

$mdadm --manage /dev/md0 --add /dev/sdb2 

Fig. 7. Adding the new disk to the RAID array. 

Replace /dev/sdb2 with the new disk's partition name.

 

Step 7: Synchronize the RAID Array 

  • Check the synchronization status: 

$cat /proc/mdstat 

Monitor the synchronization process. This may take some time depending on the size of the disks and the RAID level. 

Troubleshooting 

If you encounter any issues during these steps, refer to additional resources for more detailed information or contact technical support. 

Summary 

By following these steps, you can effectively manage your RAID array, ensuring minimal downtime and data integrity.