IBM VIO Server

From Linux on Power wiki

Contents 1 IBM Virtual I/O Server 1.1 Installation 1.2 Initial setup 1.3 Virtual I/O Server commands 1.4 Virtual SCSI setup 1.5 Virtual I/O Optical Device Library 1.6 Virtual Ethernet setup 1.6.1 Scenario A (one VIO server) 1.6.2 Scenario B (two VIO servers)

IBM Virtual I/O Server

The Virtual I/O Server is part of the IBM eServer p5 Advanced Power Virtualization hardware feature. Virtual I/O Server allows sharing of physical resources between LPARs including virtual SCSI and virtual networking. This allows more efficient utilization of physical resources through sharing between LPARs and facilitates server consolidation.

Installation

You have two options to install the AIX-based VIO Server:

1. Install from CD
2. Install from network via an AIX NIM-Server

Installation method #1 is probably the more frequently used method in a pure Linux environment as installation method #2 requires the presence of an AIX NIM (Network Installation Management) server. Both methods differ only in the initial boot step and are then the same. They both lead to the following installation screen:

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM                             IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM     STARTING SOFTWARE       IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM        PLEASE WAIT...       IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM                             IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

Elapsed time since release of system processors: 51910 mins 20 secs







-------------------------------------------------------------------------------
                       Welcome to the Virtual I/O Server.
                       boot image timestamp: 10:22 03/23
                 The current time and date: 17:23:47 08/10/2005
               number of processors: 1    size of memory: 2048MB
 boot device: /pci@800000020000002/pci@2,3/ide@1/disk@0:\ppc\chrp\bootfile.exe
SPLPAR info:    entitled_capacity: 50    platcpus_active: 2
This system is SMT enabled: smt_status: 00000007; smt_threads: 2
                     kernel size: 10481246; 32 bit kernel
-------------------------------------------------------------------------------

The next step then is to define the system console. After some time you should see the following screen:

 ******* Please define the System Console. *******

Type a 1 and press Enter to use this terminal as the
  system console.
Pour definir ce terminal comme console systeme, appuyez
  sur 1 puis sur Entree.
Taste 1 und anschliessend die Eingabetaste drücken, um
  diese Datenstation als Systemkonsole zu verwenden.
Premere il tasto 1 ed Invio per usare questo terminal
  come console.
Escriba 1 y pulse Intro para utilizar esta terminal como
  consola del sistema.
Escriviu 1 1 i premeu Intro per utilitzar aquest
  terminal com a consola del sistema.
Digite um 1 e pressione Enter para utilizar este terminal
  como console do sistema.

HARDWARE SYSTEM MICROCODE
Licensed Internal Code - Property of IBM
(C) Copyright IBM Corp. 1990, 1994.
All rights reserved.

US Government Users Restricted Rights -
Use, duplication or disclosure restricted
by GSA ADP Schedule Contract with IBM Corp.

>>>  1 Type 1 and press Enter to have English during install.
     2 Entreu 2 i premeu Intro per veure la instal\ufffflaci\uffff en catal
     3 Entrez 3 pour effectuer l'installation en fran\uffffais.
     4 Für Installation in deutscher Sprache 4 eingeben
        und die Eingabetaste drücken.
     5 Immettere 5 e premere Invio per l'installazione in Italiano.
     6 Digite 6 e pressione Enter para usar Portugu\uffffs na instala\uffff\uffffo.
     7 Escriba 7 y pulse Intro para la instalaci\uffffn en espa\uffffol.











    88  Help ?


>>> Choice [1]:

This is the main installation menu of the AIX-based VIO-Server:

                      Welcome to Base Operating System
                      Installation and Maintenance

Type the number of your choice and press Enter.  Choice is indicated by >>>.

>>> 1 Start Install Now with Default Settings

    2 Change/Show Installation Settings and Install

    3 Start Maintenance Mode for System Recovery










    88  Help ?
    99  Previous Menu

>>> Choice [1]:

                 System Backup Installation and Settings

Either type 0 and press Enter to install with the current settings, or type the
number of the setting you want to change and press Enter.

      Setting:                                 Current Choice(s):

    1 Disk(s) where you want to install ...... hdisk0
         Use Maps............................. No
    2 Shrink File Systems..................... No
    3 Import User Volume Groups............... No
    4 Recover Devices......................... No

>>> 0 Install with the settings listed above.





                       +-----------------------------------------------------
    88  Help ?         |    WARNING: Base Operating System Installation will
    99  Previous Menu  |    destroy or impair recovery of ALL data on the
                       |    destination disk hdisk0.
>>> Choice [0]:

Selection of disks:

                 Change Disk(s) Where You Want to Install

Type one or more numbers for the disk(s) to be used for installation and press
Enter. To cancel a choice, type the corresponding number and Press Enter.
At least one bootable disk must be selected. The current choice is indicated
by >>>.

        Name     Location Code    Size(MB)   VG Status   Bootable   Maps

>>>  1  hdisk0   03-08-00-3,0     70006   none            Yes    No
     2  hdisk1   03-08-00-4,0     70006   none            Yes    No
     3  hdisk2   03-08-00-5,0     70006   none            Yes    No
     4  hdisk3   03-08-00-8,0     70006   none            Yes    No



>>>  0   Continue with choices indicated above
    55  More Disk Options
    66  Devices not known to Base Operating System Installation
    77  Display More Disk Information
    88  Help ?
    99  Previous Menu

>>> Choice [0]:

Now start the installation.

                        Installing Base Operating System




        Please wait...













        Approximate     Elapsed time
     % tasks complete   (in minutes)


          8               0      1% of mksysb data restored.

 Licensed Materials - Property of IBM

 5765G0300
   (C) Copyright International Business Machines Corp. 1985, 2004.
   (C) Copyright AT&T 1984, 1985, 1986, 1987, 1988, 1989.
   (C) Copyright Regents of the University of California 1980, 1982, 1983, 1985, 1986, 1987, 1988, 1989.
   (C) Copyright BULL 1993, 2004.
   (C) Copyright Digi International Inc. 1988-1993.
   (C) Copyright Interactive Systems Corporation 1985, 1991.
   (C) Copyright ISQUARE, Inc. 1990.
   (C) Copyright Mentat Inc. 1990, 1991.
   (C) Copyright Open Software Foundation, Inc. 1989, 1994.
   (C) Copyright Sun Microsystems, Inc. 1984, 1985, 1986, 1987, 1988, 1991.

 All rights reserved.
 US Government Users Restricted Rights - Use, duplication or disclosure
 restricted by GSA ADP Schedule Contract with IBM Corp.
forced unmount of /var
Rebooting . . .

-------------------------------------------------------------------------------
                       Welcome to the Virtual I/O Server.
                       boot image timestamp: 17:44 08/10
                 The current time and date: 17:46:01 08/10/2005
               number of processors: 1    size of memory: 2048MB
    boot device: /pci@800000020000002/pci@2,4/pci1069,b166@1/scsi@0/sd@3:2
SPLPAR info:    entitled_capacity: 50    platcpus_active: 2
This system is SMT enabled: smt_status: 00000007; smt_threads: 2
                     kernel size: 12405120; 64 bit kernel
-------------------------------------------------------------------------------

Saving Base Customize Data to boot disk
Starting the sync daemon
Starting the error daemon
System initialization completed.
Setting tunable parameters...complete
Starting Multi-user Initialization
 Performing auto-varyon of Volume Groups
 Activating all paging spaces
0517-075 swapon: Paging device /dev/hd6 is already active.
swapon: Paging device /dev/paging00 activated.

The current volume is: /dev/hd1
Primary superblock is valid.

The current volume is: /dev/hd10opt
Primary superblock is valid.
 Performing all automatic mounts
Multi-user initialization completed
System reconfiguration in progress.  Please wait.

Rebuild of Scriptinfo file is complete

Checking for srcmstr active...complete
Starting tcpip daemons:
0513-059 The syslogd Subsystem has been started. Subsystem PID is 217224.
Aug 10 12:46:40 localhost syslog:err|error syslogd: /tmp/syslog.out: errno = 2
0513-059 The sendmail Subsystem has been started. Subsystem PID is 180374.
0513-059 The portmap Subsystem has been started. Subsystem PID is 69844.
0513-059 The inetd Subsystem has been started. Subsystem PID is 118936.
0513-059 The snmpd Subsystem has been started. Subsystem PID is 184462.
0513-059 The hostmibd Subsystem has been started. Subsystem PID is 131306.
0513-059 The snmpmibd Subsystem has been started. Subsystem PID is 172216.
0513-059 The aixmibd Subsystem has been started. Subsystem PID is 204952.
0513-059 The muxatmd Subsystem has been started. Subsystem PID is 164006.
Finished starting tcpip daemons.
Starting NFS services:
0513-059 The biod Subsystem has been started. Subsystem PID is 176302.
0513-059 The rpc.lockd Subsystem has been started. Subsystem PID is 262390.
0513-059 The ctrmc Subsystem has been started. Subsystem PID is 229540.
Completed NFS services.
IBM Virtual I/O Server

login:

Initial setup

After the reboot you are presented with the VIO-Server login prompt. You can't login as user root as you have to use the special user id padmin. No initial default password is set. Immediately after login you are forced to set a new password.

[compat]: 3004-610 You are required to change your password.
        Please choose a new one.

padmin's New password:
Enter the new password again:


The following file has been updated: .profile
Changes will take affect at next login.

Before you can do anything you have to accept the I/O Server license. This is done with the license command.

$ lsmap
The I/O Server license must be accepted before running this command.
Only the chlang command may be run before accepting the license.
To view the license, run the license command with the -view option.
If the -lang flag is not specified the license will by displayed in
the current locale (set by the chlang command).  If the license is not
available in the language selected, English will be the default.
To accept the license, run the license command with the -accept option.

Locale currently set to: "en_US".

Usage: license [-view] [-accept] [-lang Name]
       license [-ls]

View and accept the I/O Server license agreement.

        -accept Accept the license agreement.
        -lang   Specifies the language-territory (locale name) the license
                is displayed in.
        -ls     Lists available locales.
        -view   Displays the license agreement.


$

Please enter:

license -accept

Before you start to define and assign resources please make sure you have the latest version and updates installed before you proceed. Updates and support for the AIX-based Virtual I/O Server can be obtained at: http://techsupport.services.ibm.com/server/vios/home.html

Details beyond the scope of this publications can be found in the following IBM Redbook Advanced POWER Virtualization on IBM eServer p5 Servers: Introduction and Basic Configuration at http://www.redbooks.ibm.com/abstracts/sg247940.html?Open.

Virtual I/O Server commands

Once you are logged in as user padmin you find yourself in a restricted Korn shell with only a limited set of commands. You can see all available commands with the command help. All these commands are shell aliases to a single SUID-binary called ioscli which is located in the directory /usr/ios/cli/bin. If you are familiar with AIX you will recognize most commands but most command line parameters differ from the AIX versions.

As there are no man pages available you can see all options for each command separately by issueing the command help <command>. Here is an example for the command lsmap:

$ help lsmap
Usage: lsmap {-vadapter ServerVirtualAdapter | -plc PhysicalLocationCode |
             -all}
             [-net] [-fmt delimiter]
       Displays the mapping between physical and virtual devices.

       -all         Displays mapping for all the server virtual adapter
                    devices.

       -vadapter    Specifies the server virtual adapter device
                    by device name.

       -plc         Specifies the server virtual adapter device
                    by physical location code.

       -net         Specifies supplied device is a virtual server
                    Ethernet adapter.

       -fmt         Divides output by a user-specified delimiter.

A very important command is oem_setup_env which gives you access to the regular AIX command line interface. This is provided solely for the installation of OEM device drivers.

Only the commands necessary for the setup of our defined scenarios (see chapter XXX) will be described next. Details beyond the scope of this publications can be found in the following IBM Redbook Advanced POWER Virtualization on IBM eServer p5 Servers: Introduction and Basic Configuration at http://www.redbooks.ibm.com/abstracts/sg247940.html?Open.

Virtual SCSI setup

To map a LV

• mkvg: creates the volume group, where a new LV will be created using the mklv command
• lsdev: shows the virtual SCSI server adapters that could be used for mapping with the LV
• mkvdev: maps the virtual SCSI server adapter to the LV
• lsmap -all: shows the mapping information

To map a physical disk

• lsdev: shows the virtual SCSI server adapters that could be used for mapping with a physical disk
• mkvdev: maps the virtual SCSI server adapter to a physical disk
• lsmap -all: shows the mapping information

Client partition commands

• No commands needed, the Linux kernel is notified immediately

Create new volume group datavg with member disk hdisk1

• mkvg -vg datavg hdisk1

Create new logical volume vdisk0 in volume group

• mklv -lv vdisk0 datavg 10G

Maps the virtual SCSI server adapter to the logical volume

• mkvdev -vdev vdisk0 -vadapter vhost0

Display the mapping information

• lsmap -all

Virtual I/O Optical Device Library

To keep from having to load and unload physical media into a CD or DVD drive, the media can be copied into a .iso file. Multiple virtual CD drives can be created and that .iso file can be "loaded" into as many virtual CD drives simultaneously to assign to any number of LPARs. The over all process is to first create a filesystem to hold the virtual CDs, create the .iso images, create virtual CD devices, load the media, run cfgmgr and mount the CD drive on the client LPAR. The process of removing the virtual CD is to first unmount and remove the cd device on the client LPAR, unload the virtual CD device, and then remove the virtual device itself.

Create the VMLibrary Logical Volume and Filesystem

• mkrep -sp rootvg -size 16G

Create a disk image of a CD or DVD loaded in the CD drive

• mkvopt -name disk_image_name.iso -dev cd0 -ro

Create the Virtual CD Device on the VIO Server

• mkvdev -fbo -vadapter vhost0 -dev hostnamecd0

Load Virtual CD Device on the VIO Server

• loadopt -vtd hostnamecd0 -disk disk_image_name.iso

Verify the device is loaded with lsmap

• lsmap -vadapter vhost0

Configure the CD Device on the client LPAR

• cfgmgr
• lsdev | grep cd

Mount the virtual CD on the client LPAR

• mkdir /cdrom
• mount -v'cdrfs' -r /dev/cd0 /cdrom

Use and access the mounted CD as you would a normal CD/DVD drive.

Unmount and remove the CD on the client LPAR

• rmdev -dl cd0

Unload the Virtual CD Drive on the VIO server

• unloadopt -vtd hostnamecd0

Remove the Virtual CD Device from the VIO server

• rmvdev -vtd hostnamecd0

Virtual Ethernet setup

To list all virtual and physical adapters use the lsdev -type adapter command.

$ lsdev -type adapter
name            status     description

ent0            Available  2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
ent1            Available  2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
ent2            Available  Virtual I/O Ethernet Adapter (l-lan)
ide0            Available  ATA/IDE Controller Device
sisscsia0       Available  PCI-X Dual Channel Ultra320 SCSI Adapter
vhost0          Available  Virtual SCSI Server Adapter
vhost1          Available  Virtual SCSI Server Adapter
vhost2          Available  Virtual SCSI Server Adapter
vhost3          Available  Virtual SCSI Server Adapter
vsa0            Available  LPAR Virtual Serial Adapter

Choose the virtual Ethernet adapter we want to map to the physical Ethernet adapter.

$ lsdev -virtual
name            status     description

ent2            Available  Virtual I/O Ethernet Adapter (l-lan)
vhost0          Available  Virtual SCSI Server Adapter
vhost1          Available  Virtual SCSI Server Adapter
vhost2          Available  Virtual SCSI Server Adapter
vhost3          Available  Virtual SCSI Server Adapter
vsa0            Available  LPAR Virtual Serial Adapter

The command mkvdev maps a physical adapter to a virtual adapter, creates a layer 2 network bridge and defines the default virtual adapter with its default VLAN ID. It creates a new Ethernet interface, e.g., ent3.

Make sure the physical and virtual interfaces are unconfigured (down or detached).

Scenario A (one VIO server)

Create a shared ethernet adapter ent3 with a physical one (ent0) and a virtual one (ent2) with PVID 1:

$ mkvdev -sea ent0 -vadapter ent2 -default ent2 -defaultid 1
ent3 Available
en3
et3

This has created a new shared ethernet adapter ent3 (you can verify that with the lsdev command). Now configure the TCP/IP settings for this new shared ethernet adapter (ent3). Please note that you have to specify the interface (en3) and not the adapter (ent3).

$ mktcpip -hostname op710-1-vio -inetaddr 9.156.175.231 -interface en3 \
          -netmask 255.255.255.0 -gateway 9.156.175.1 -nsrvaddr 9.64.163.21 \
          -nsrvdomain munich.de.ibm.com

Scenario B (two VIO servers)

Create a shared ethernet adapter ent3 with a physical one (ent0) and a virtual one (ent2) with PVID 1:

mkvdev -sea ent0 -vadapter ent2 -default ent2 -defaultid 1

Configure the TCP/IP settings for the new shared ethernet adapter (ent3):

mktcpip -hostname op710-1-vio -inetaddr 9.156.175.231 -interface en3 \
        -netmask 255.255.255.0 -gateway 9.156.175.1 -nsrvaddr 9.64.163.21 \
        -nsrvdomain munich.de.ibm.com

Client partition commands

• No new commands needed just the typical TCP/IP configuration is done on the virtual Ethernet interface that it is defined in the client partition profile on the HMC

---

• See http://www-128.ibm.com/developerworks/eserver/library/es-aix-vioserver-v2/ for instructions how to set up IBM VIO Server.

Trick:

dsh -l padmin -n VIO_Server1,VIO_Server21 "echo fget_config -Av | ioscli oem_setup_env"