From Linux on Power wiki

Contents 1 Configuring Software Raid and LVM on Power using Red Hat Enterprise Linux Advanced Server 4U2 Update 2 2 Prerequisites 3 Background 4 Target layout 4.1 Partitioning layout 4.1.1 Raid layout 4.1.2 LVM layout 4.2 Step by step: manual installation 4.2.1 Start installation 4.2.2 Leave anaconda installer 4.2.3 Command line partitioning 4.2.3.1 Create partitions on /dev/sda (first disk) 4.2.3.2 Create partitions on /dev/sdb (second disk) 4.2.4 Create Raid devices 4.2.5 Create volume group and logical volumes 4.2.6 Get back to anaconda installer 4.3 Postinstallation tasks 4.3.1 Mirror PrepBoot partition 4.3.2 Set the boot list 4.4 References: 5 After Update to U3 or U4

Configuring Software Raid and LVM on Power using Red Hat Enterprise Linux Advanced Server 4U2 Update 2

Tomas Baublys baublys AT de.ibm.com

Prerequisites

We are using Red Hat Enterprise Linux Advanced Server 4 (RHEL AS4) on OpenPower 720. It is recommended to use hardware raid adapters on all server systems. Still there are cases, where hardware raid is not applicable: for example for a scenario using two VIO servers serving each one disk to the client LPARs and using software mirroring on the client side.

Background

Using software RAID on POWER is different from using it on Intel due to the different boot loader. Power uses the Yaboot1 boot loader which is loaded by Open Firmware2 after identifying the PReP boot partition on the disk. Only disks containing the PReP boot partition can be selected in SMS (user interface to the Open Firmware). Yaboot starts and scans all the primary disk partitions searching for its own configuration file (usually /etc/yaboot.conf) and the kernel image to load. Unfortunately Yaboot can not scan software mirrored md-devices or LVM logical volumes. It also contains a build in knowledge of only some file systems: ext2 , ext3, reiserfs and fat. We are going to use separate /boot partition which will be a second primary partition on both disks and contain boot kernel, initrd and etc directory with yaboot.conf inside.

Target layout

Partitioning layout

We are going to create following partitions on our first disk:

Name	Type	Format	Size	Purpose
/dev/sda1	PrePBoot / 41	No	6 MB < 10MB	PrepBoot partition, loaded by firmware
/dev/sda2	Linux Raid /fd	EXT3	100 MB	Raid device for /boot, will be used for md0
/dev/sda3	Linux Raid /fd	No	16 GB (all place you have left)	Raid device for LVM, will be used for md1

We will create almost the same partitions on the second disk, the only difference is, that the second disk should not have a PrePBoot partition. We will change the type later.

Name	Type	Format	Size	Purpose
/dev/sdb1	Linux 83 /later PrePBoot	No	6 MB < 10MB	second PrepBoot partition
/dev/sdb2	Linux Raid /fd	EXT3	100 MB	Boot partition. will be used for md0
/dev/sdb3	Linux Raid /fd	No	10 GB (all place you have left)	Raid device for LVM, will be used for md1

Raid layout

We will create a raid device /dev/md0 containing /dev/sda2 and /dev/sdb2 and a raid device /dev/md1 containing /dev/sda3 and /dev/sdb3

Name	Include	Format	Mount point	Purpose
/dev/md0	/dev/sda2, /dev/sdb2	ext3		/boot
/dev/md1	/dev/sda3, /dev/sdb3	LVM		LVM

LVM layout

We will use our raid device /dev/md0 as a physical disk and include it in the volume group /dev/volgrp01. We will create following logical volumes in this volume group.

Name	Format	Size	Mount point
usrlv	ext3 or reiserfs	3 GB	/usr
optlv	ext3 or reiserfs	1 GB	/opt
varlv	ext3 or reiserfs	512 MB	/var
tmplv	ext3 or reiserfs	512 MB	/tmp
swaplv	swap	1 GB	no

Step by step: manual installation

Start installation

Start your installation from CD or network. Power on your system or activate your LPAR and press 1 to enter SMS menu. Press 5 to enter “Select Boot Options” section, 1 in the next screen select “Select Install or Boot device” and select the boot device you are going to use.

Leave anaconda installer

After anaconda installer starts you will get the greeting screen and stop at “Disk partioning setup” screen.

Unfortunately we can not use Autopartition (it would setup LVM without any mirroring). Also the “Disk Druid” was not usable for a proper raid configuration. We are going to leave anaconda installer for a while and partition our disks manualy. To do that press CTRL+z and you will get a shell.

Command line partitioning

Create partitions on /dev/sda (first disk)

• Use fdisk /dev/sda command. Use “h” for help, “n” for new partition, “t” for type, “a” to set boot flag, and “w” to write your partitioning to the disk.

• Create following partitions:

Disk /dev/sda: 18.2 GB, 18253611008 bytes
64 heads, 32 sectors/track, 17408 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          10       10224   41  PPC PReP Boot
/dev/sda2              11         106       98304   fd  Linux raid autodetect
/dev/sda3             107       17408    17717248   fd  Linux raid autodetect

Create partitions on /dev/sdb (second disk)

In order to create the same partioning layout on the second disk we can repeat the same steps above or use sfdisk instead fdisk.

• Run “sfdisk -d /dev/sda > /tmp/sda.out” command in order to dump the partition table in to a file.

• Run “sfdisk /dev/sdb < /tmp/sda.out” in order to creat identical partition table on the second disk.
• Set the partition type to Linux/83 for /dev/sdb1 in order to avoid yaboot confusion.

Create Raid devices

Use following command in order to create a raid-device /dev/md0 containing the partitions /dev/sda2 and /dev/sdb2 for /boot:

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2

and for LVM:

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3

Create volume group and logical volumes

In order to configure LVM enter the lvm shell by issuing lvm command.

• Create physical volume:

lvm> pvcreate /dev/md1

• Create volume group:

lvm> vgcreate volgrp01 /dev/md1

• Create logical volumes:

lvm> lvcreate -L 1000 --name rootlv volgrp01
lvm> lvcreate -L 4000 --name usrlv volgrp01
lvm> lvcreate -L 512  --name tmplv volgrp01
lvm> lvcreate -L 512  --name homelv volgrp01
lvm> lvcreate -L 512  --name optlv volgrp01
lvm> lvcreate -L 512  --name varlv volgrp01
lvm> lvcreate -L 512  --name swap volgrp01
lvm> quit

Get back to anaconda installer

Type “exit” on the shell prompt and you will get back to anaconda installer. Select to use Disk Druid for partitioning. Select your logical volumes and press F3 for edit. In the next screen choose file system options and set the volumes to be formatted.

One screen back set the proper mount point for each volume:

Select /dev/md0 and set it to be formatted and mounted as /boot. After finishing continue your installation, and let the system reboot.

Postinstallation tasks

If everything is gone well, your system should reboot and come up with software raid /dev/md0 and /dev/md1 and LVM on it. If your system does not boot, the most likely cause will be the size of the PrepBoot partition. Make sure it is not bigger then 10 MB. You may also need to set your boot device in SMS menu.

Your partitioning should look like this:

[root@op720-1-client2 ~]# fdisk -l

Disk /dev/sda: 18.2 GB, 18253611008 bytes
64 heads, 32 sectors/track, 17408 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          10       10224   41  PPC PReP Boot
/dev/sda2              11         106       98304   fd  Linux raid autodetect
/dev/sda3             107       17408    17717248   fd  Linux raid autodetect

Disk /dev/sdb: 18.2 GB, 18253611008 bytes
64 heads, 32 sectors/track, 17408 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          10       10224   41  PPC PReP Boot
/dev/sdb2              11         106       98304   fd  Linux raid autodetect
/dev/sdb3             107       17408    17717248   fd  Linux raid autodetect

And your logical volumes:

[root@op720-1-client2 ~]# lvscan
 ACTIVE            '/dev/volgrp01/rootlv' [1000.00 MB] inherit
 ACTIVE            '/dev/volgrp01/usrlv' [3.91 GB] inherit
 ACTIVE            '/dev/volgrp01/tmplv' [1000.00 MB] inherit
 ACTIVE            '/dev/volgrp01/optlv' [1000.00 MB] inherit
 ACTIVE            '/dev/volgrp01/varlv' [1000.00 MB] inherit
 ACTIVE            '/dev/volgrp01/swaplv' [1000.00 MB] inherit

And you mount points like this:

[root@op720-1-client2 ~]# mount
/dev/mapper/volgrp01-rootlv on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/md0 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/mapper/volgrp01-optlv on /opt type ext3 (rw)
/dev/mapper/volgrp01-tmplv on /tmp type ext3 (rw)
/dev/mapper/volgrp01-usrlv on /usr type ext3 (rw)
/dev/mapper/volgrp01-varlv on /var type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

Mirror PrepBoot partition

We need to have the same working PrepBoot partition on the second disk in order to be able to boot if the first one is absent or broken.

• We are going to mirror the PrepBoot partition using dd:

dd if=/dev/sda1 of=/dev/sdb1

• Now we set the boot flag:

parted /dev/sdb set 1 boot on

Set the boot list

We also need to set the OpenFirmware boot list to contain both disks:

bootlist -m normal sda sdb
bootlist -m normal -o
sda
sdb

References:

1. Information on basic system setup http://www.ibm.com/developerworks/l-pow-opsetup
2. Linux on Power wiki page http://oss.gonicus.de/openpower
3. Disk mirroring using software RAID on RHEL 3.0 AS for IBM pSeries http://www-106.ibm.com/developerworks/eserver/library/es-SW_RAID_LINUX.html

Thanks to: Brian J. King, René Bersier, Michael Perzl, Dustin Kirkland

____

After Update to U3 or U4

If you did updated your system to U3 or U4 you can use this small script to convert your current /boot to the sw-raid device

#!/bin/sh
clear
VERSION=$(grep "Update" /etc/redhat-release|rev |cut -c 2)
if [ $VERSION = 4 ] ; then
echo You have an update to version $VERSION
else
echo "*******************************************************"
echo "*******************************************************"
echo "**                                                   **"
echo "**  Please Update to U4 first (edit script for U3)   **"
echo "**    Script execution automatically canceled        **"
echo "**                                                   **"
echo "*******************************************************"
echo "*******************************************************" 
exit
fi
#checking for md31
cat /proc/mdstat |grep md31
if [ $? = 0 ] ; then
echo "*******************************************************"
echo "*******************************************************"
echo "**                                                   **"
echo "**    /dev/md31 already present on the system        **"
echo "**     Script execution automatically canceled       **"
echo "**                                                   **"
echo "*******************************************************"
echo "*******************************************************"
exit
fi 
echo "*******************************************************"
echo "**                                                   **"
echo "**                                                   **"
echo "**  Press Ctrl+c to cancel. Waiting 3 seconds        **"
echo "**                                                   **"
echo "**                                                   **"
echo "*******************************************************" 
sleep 6 
# Mount, backup and copy /boot
mount /dev/sda2 /boot
tar czf /root/boot.tgz /boot
cp -a /boot /tmp
# umount boot and boot1
umount /dev/sda2
umount /dev/sdb2
# Delete all labels
tune2fs -L "" /dev/sda2
tune2fs -L "" /dev/sdb2 
# Change partition type
sfdisk --change-id /dev/sda 2 fd
sfdisk --change-id /dev/sdb 2 fd 
# Creating  md31
mdadm -C -f -R /dev/md31 -l 1 -n 2 /dev/sda2 /dev/sdb2
# Format
mkfs.ext3 /dev/md31
#  Applying changes to  /etc/fstab
cp /etc/fstab /etc/fstab.031
sed -e '/boot/d' /etc/fstab.031 > /etc/fstab
echo "/dev/md31         /boot                   ext3    defaults        1 2" >> /etc/fstab 
# Mount
mount /dev/md31 /boot
# Copy boot content back
cp -a /tmp/boot/* /boot