From Linux on Power wiki

Contents 1 Configuring Software Raid and LVM on Power using Red Hat Enterprise Linux Advanced Server 4 1.1 See recent Version of this document at http://www-941.ibm.com/collaboration/wiki/display/LinuxP/Software+Raid+Setup+Red+Hat 2 Prerequisites 3 Background 4 Target layout 4.1 Partitioning layout 4.1.1 Raid layout 4.1.2 LVM layout 4.2 Step by step: manual installation 4.2.1 Start installation 4.2.2 Leave anaconda installer 4.2.3 Command line partitioning 4.2.3.1 Create partitions on /dev/sda (first disk) 4.2.3.2 Create partitions on /dev/sdb (second disk) 4.2.4 Create Raid device 4.2.5 Create volume group and logical volumes 4.2.6 Get back to anaconda installer 4.3 Postinstallation tasks 4.3.1 Mirror PrepBoot partition 4.3.2 Set the boot list 4.3.3 Mirror /boot partition 4.3.4 Remove /boot from /etc/fstab 5 Kickstart configuration 5.1 References:

Configuring Software Raid and LVM on Power using Red Hat Enterprise Linux Advanced Server 4

Tomas Baublys baublys AT de.ibm.com

See recent Version of this document at http://www-941.ibm.com/collaboration/wiki/display/LinuxP/Software+Raid+Setup+Red+Hat

Prerequisites

We are using Red Hat Enterprise Linux Advanced Server 4 (RHEL AS4) on OpenPower 720. It is recommended to use hardware raid adapters on all server systems. Still there are cases, where hardware raid is not applicable: for example for a scenario using two VIO servers serving each one disk to the client LPARs and using software mirroring on the client side.

Background

Using software RAID on POWER is different from using it on Intel due to the different boot loader. Power uses the Yaboot1 boot loader which is loaded by Open Firmware2 after identifying the PReP boot partition on the disk. Only disks containing the PReP boot partition can be selected in SMS (user interface to the Open Firmware). Yaboot starts and scans all the primary disk partitions searching for its own configuration file (usually /etc/yaboot.conf) and the kernel image to load. Unfortunately Yaboot can not scan software mirrored md-devices or LVM logical volumes. It also contains a build in knowledge of only some file systems: ext2 , ext3, reiserfs and fat. We are going to use separate /boot partition which will be a second primary partition on both disks and contain boot kernel, initrd and etc directory with yaboot.conf inside.

Target layout

Partitioning layout

We are going to create following partitions on our first disk:

Name	Type	Format	Size	Purpose
/dev/sda1	PrePBoot / 41	No	6 MB < 10MB	PrepBoot partition, loaded by firmware
/dev/sda2	Linux 83	EXT3	100 MB	Faked boot partition
/dev/sda3	Linux Raid /fd	No	10 GB (all place you have left)	Raid device for LVM, will be used for md1

We will create almost the same partitions on the second disk, the only difference is, that the second disk should not have a PrePBoot partition. We will change the type later.

Name	Type	Format	Size	Purpose
/dev/sdb1	Linux 83 /later PrePBoot	No	6 MB < 10MB	second PrepBoot partition
/dev/sdb2	Linux 83	EXT3	100 MB	Faked boot partition
/dev/sdb3	Linux Raid /fd	No	10 GB (all place you have left)	Raid device for LVM, will be used for md1

Raid layout

We will create a raid device /dev/md0 containing /dev/sda3 and /dev/sdb3

Name	Include	Format	Mount point	Purpose
/dev/md0	/dev/sda3, /dev/sdb3	LVM		LVM

LVM layout

We will use our raid device /dev/md0 as a physical disk and include it in the volume group /dev/volgrp01. We will create following logical volumes in this volume group.

Name	Format	Size	Mount point
usrlv	ext3 or reiserfs	1 GB	/usr
optlv	ext3 or reiserfs	1 GB	/opt
varlv	ext3 or reiserfs	512 MB	/var
tmplv	ext3 or reiserfs	512 MB	/tmp
swaplv	swap	1 GB	no

Step by step: manual installation

Start installation

Start your installation from CD or network. Power on your system or activate your LPAR and press 1 to enter SMS menu. Press 5 to enter “Select Boot Options” section, 1 in the next screen select “Select Install or Boot device” and select the boot device you are going to use.

Leave anaconda installer

After anaconda installer starts you will get the greeting screen and stop at “Disk partioning setup” screen.

Unfortunately we can not use Autopartition (it would setup LVM without any mirroring). Also the “Disk Druid” was not usable for a proper raid configuration. We are going to leave anaconda installer for a while and partition our disks manualy. To do that press CTRL+z and you will get a shell.

Command line partitioning

Create partitions on /dev/sda (first disk)

• Use fdisk /dev/sda command. Use “h” for help, “n” for new partition, “t” for type, “a” to set boot flag, and “w” to write your partitioning to the disk.

• Create following partitions:

Disk /dev/sda: 10240 cylinders, 64 heads, 32 sectors/track

  Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+      0       1-      6138   41  PPC PReP Boot
/dev/sda2          2      18      17     104873   83  Linux
/dev/sda3         19    1698    1680   10363920   fd  Linux raid autodetect

Create partitions on /dev/sdb (second disk)

In order to create the same partioning layout on the second disk we can repeat the same steps above or use sfdisk instead fdisk.

• Run “sfdisk -d /dev/sda > /tmp/sda.out” command in order to dump the partition table in to a file.

• Run “sfdisk /dev/sdb < /tmp/sda.out” in order to creat identical partition table on the second disk.
• Set the partition type to Linux/83 for /dev/sdb1 in order to avoid yaboot confusion.

Create Raid device

Use following command in order to create a raid-device /dev/md0 containing the partitions /dev/sda3 and /dev/sdb3:

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3

Create volume group and logical volumes

In order to configure LVM enter the lvm shell by issuing lvm command.

• Create physical volume:

lvm> pvcreate /dev/md0

• Create volume group:

lvm> vgcreate volgrp01 /dev/md0 

• Create logical volumes:

lvm> lvcreate -L 1000 --name rootlv volgrp01
lvm> lvcreate -L 2000 --name usrlv volgrp01
lvm> lvcreate -L 512  --name tmplv volgrp01
lvm> lvcreate -L 512  --name homelv volgrp01
lvm> lvcreate -L 512  --name optlv volgrp01
lvm> lvcreate -L 512  --name varlv volgrp01
lvm> lvcreate -L 512  --name swap volgrp01
lvm> quit

Get back to anaconda installer

Type “exit” on the shell prompt and you will get back to anaconda installer. Select to use Disk Druid for partitioning. Select your logical volumes and press F3 for edit. In the next screen choose file system options and set the volumes to be formatted.

One screen back set the proper mount point for each volume:

Select /dev/sda2 and set it to be formatted and mounted as /boot. After finishing continue your installation, and let the system reboot. Let /dev/sdb2 unformatted and without any mount point.

Postinstallation tasks

If everything is gone well, your system should reboot and come up with software raid /dev/md0 and LVM on it. If your system does not boot, the most likely cause will be the size of the PrepBoot partition. Make sure it is not bigger then 10 MB.

Mirror PrepBoot partition

We need to have the same working PrepBoot partition on the second disk in order to be able to boot if the first one is absent or broken.

• We are going to mirror the PrepBoot partition using dd:

dd if=/dev/sda1 of=/dev/sdb1

• Now we set the boot flag:

parted /dev/sdb set 1 boot on

Set the boot list

We also need to set the OpenFirmware boot list to contain both disks:

bootlist -m normal sda sdb
bootlist -m normal -o
sda
sdb

Mirror /boot partition

We also can use dd for /boot (/dev/sda2 /dev/sdb2 mirroring) or alternatively format it and just copy the contents.

mkdir /boot1 
mkfs.ext3 /dev/sdb2
mount /dev/sdb2 /boot1
cp -a /boot/* /boot1/

Remove /boot from /etc/fstab

Our /etc/fstab has a line for /boot partition. We need to remove this line (uncomment it) in order be able to boot without /dev/sda. Removing is just a quick and dirty workaround, if you have a better idea, drop me a note. The /boot file system does not really need to be mounted. You only should mount it, if you are running a kernel update. Do not forget to mirror it to /dev/sdb2.

/etc/fstab:
#LABEL=/boot             /boot                   ext2    defaults        1 2

Kickstart configuration

Here is an example of the kickstart partitioning. Unfortunately anaconda changes the order of the partitions on the second disk, so you will need to use pre or post scripts to do partitioning or correct the partitioning on the second disk.

bootloader --location=partition --append="console=hvc0 rhgb quiet"
clearpart --all --initlabel --drives=sda,sdb
part prepboot --fstype "PPC PReP Boot" --size=4 --ondisk=sda --asprimary
part /prep2 --fstype "ext3" --size=4 --ondisk=sdb --asprimary
part /boot --fstype "ext3" --size=100 --ondisk=sda --asprimary
part /boot1 --fstype "ext3" --size=100 --ondisk=sdb --asprimary
part raid.01 --size=0 --grow --ondisk=sda
part raid.02 --size=0 --grow --ondisk=sdb
raid pv.3 --level=1 --device=md0 --fstype "physical volume (LVM)" raid.01 raid.02
volgroup volgrp01 --pesize=4096 pv.3
logvol swap --fstype swap --size=512 --name=swap --vgname=volgrp01
logvol / --fstype ext3 --size=2000 --name=rootlv --vgname=volgrp01
logvol /usr --fstype ext3 --size=1000 --name=usrlv --vgname=volgrp01
logvol /var --fstype ext3 --size=512 --name=varlv --vgname=volgrp01
logvol /tmp --fstype ext3 --size=512 --name=tmplv --vgname=volgrp01
logvol /opt --fstype ext3 --size=1000 --name=optlv --vgname=volgrp01

____

References:

1. Information on basic system setup http://www.ibm.com/developerworks/l-pow-opsetup
2. Linux on Power wiki page http://oss.gonicus.de/openpower
3. Disk mirroring using software RAID on RHEL 3.0 AS for IBM pSeries http://www-106.ibm.com/developerworks/eserver/library/es-SW_RAID_LINUX.html

Thanks to: Brian J. King, René Bersier, Michael Perzl, Dustin Kirkland