Creating, viewing and installing Oracle Solaris 10 into a domain

Check out the proper commands to create a guest domain, view its devices and install Oracle Solaris 10 into it while monitoring the domain's activity.

Solution provider’s takeaway: Use the commands in this chapter excerpt to create, view, install, network and manage Logical Domains on a CMT server, create guest domains, and install Oracle Solaris 10 as a virtual machine for your customers. The Logical Domains technology lets the system administrator change the CPU and virtual I/O resources made available to guest domains, and allows you to easily clone systems. 

3.4 Installing Logical Domains and Building a Guest Domain
This section walks through the process of installing Logical Domains on a CMT server, creating a guest domain, and installing Oracle Solaris on it. The installed domain will then be set up as a master image for cloning further domains.

3.4.1 Verifying and Installing Firmware
All CMT servers and Blade Modules ship with firmware support for Logical Domains. Nevertheless, it is important to ensure that the current firmware for the server is installed, matching it to the version of Solaris running in the control domain and the version of Logical Domains management software to be installed. Current information can be obtained from The Release Notes, Reference Manual, and Administration Guide for Logical Domains software list the firmware levels needed on each server model. Use the instructions in the server’s Installation Guide or Administration Guide for verifying and installing the firmware.

3.4.2 Installing Logical Domains Software
Examples in this section are taken from Logical Domains version 1.2. (Note: File names, command syntax, and screen output may vary slightly in future releases.) Logical Domains software can be downloaded from coolthreads/ldoms/. The software is delivered as a .zip file, which can be unpacked by issuing the following command (or something similar):

# unzip

Before installing the software, read the README file contained in the opened .zip file. It will describe any prerequisite patches to install. If any patches are required, download them from and install them using patchadd(1M) as directed.

At that point, you can run the install script, responding to any prompts as needed.

You can then verify that the software is installed and that the Logical Domains daemon is running. The command-line interface is accessed by issuing the ldm command. At this time, a single domain will be running on the server that owns all of the installed memory and CPUs.

The following example is taken from a small (6-core) T1000. The warning that the system is in configuration mode is displayed after each ldm command until the system is rebooted with the new configuration. For brevity, we will omit the subsequent identical warning messages.

3.4.3 Configuring the Control Domain
Because the control domain initially owns all hardware resources, it must be resized to provide the RAM and CPUs to be allocated to guest domains. Essential services for providing virtual disk, network, and consoles should be defined now as well. When the domain is rebooted, it will have the specified number of CPUs and amount of RAM. The remainder of those resources will be available for allocation to guest domains. The sequence will look similar to this sequence:

Line 1 defines a virtual disk server. Notice the naming convention: The name of the server is primary-vds0, indicating that the service operates in the domain named primary (the control domain) and is the initial virtual disk server (vds0). The last token on the line indicates which domain will run this service. While following this convention is not necessary, it is highly recommended, as the name of the service makes its function self-documenting. Note also that this naming convention is separate from Solaris device naming.

Line 2 defines a vcc that will listen for local connections on ports 5000 to 5100. Line 3 defines a virtual Layer 2 switch, primary-vsw0, which is associated with the physical NIC device e1000g0. Multiple virtual switches can be defined and attached to different NIC devices or to no NIC device at all. Lines 4, 5, and 6 describe the control domain: It has a single cryptographic accelerator, 4 CPUs, and 3 GB of RAM. The remaining lines save this initial configuration in firmware so it will persist after a power cycle and then reboot the server.

3.4.4 Network Connectivity Between Primary and Guest Domains
By default, networking on a virtual switch connecting the control domain and guest domains is disabled. This approach provides an additional layer of security by isolating the control domain from guest domains’ network traffic. If this situation is not desired—for example, if the control domain is to be used as a JumpStart server for guest domains—the virtual switch can be configured as a network device and may then be used as the primary interface instead of the physical device to which the switch is assigned.

To configure the virtual switch as a network device, first issue the command ifconfig -a to get all the network parameters for the physical device (in the following example, e1000g0). Then unplumb the device, and replumb the virtual switch (in this example, vsw0) with the same information. When this procedure is complete, guest domains will be able to communicate with the control domain via this network connection.

Important Note!
Before you attempt to reconfigure the virtual switch, you must log in from the control domain’s console or from a different interface than the interface being unplumbed; otherwise, you will abruptly terminate your session! Terminating the network interface you are using for your login session can be embarrassing.

This change does not persist over a control domain reboot, so you must update /etc/hostname.* and /etc/dhcp/* to make it permanent. In this example, you would rename the configuration files for e1000g0 to the equivalent files for vsw0 by issuing the following commands:

# mv /etc/hostname.e1000g0 /etc/hostname.vsw0
# mv /etc/dhcp.e1000g0 /etc/dhcp.vsw0

3.4.5 Creating a Domain and Installing Oracle Solaris
In the following example, we create a guest domain imaginatively named ldom1, with no cryptographic accelerator, a single virtual network device, and a virtual disk residing in a ZFS file system. The following commands define a ZFS file system and allocate within it an empty 10 GB file that will be used for the Solaris system disk:

# zfs create rpool/ldoms
# zfs set mountpoint=/ldoms rpool/ldoms
# zfs set compression=on rpool/ldoms
# zfs create rpool/ldoms/ldom1
# mkfile -n 10g /ldoms/ldom1/disk0.img

In the preceding lines, ZFS compression is turned on to save disk space. The option to mkfile creates an empty file: No disk blocks are allocated until data is written to them. The file takes up no disk space, even though it is apparently 10 GB in size:

The virtual disk can also be allocated from other back-ends, such as a physical disk, but ZFS provides operational flexibility.

The following commands create the domain. Lines 2 and 3 set the number of CPUs and cryptographic accelerators. Line 4 sets the amount of RAM assigned to the domain. Line 5 creates the virtual network device using the virtual Layer 2 switch defined previously. Additional network devices could be defined if we wanted the domain to reside on separate networks. Line 6 exports the empty disk image as a virtual volume vol10@primary-vds0 from the virtual disk service. Line 7 imports this volume as a virtual disk vdisk10 into the guest domain. The commands for adding a virtual disk are a bit more complicated than the others; they can be interpreted as first defining a resource exported by the virtual disk server and then importing that resource into the domain that uses it. Finally, lines 8 and 9 do the same for a file containing the ISO format image of an Oracle Solaris installation DVD.

At this point, the domain’s definition is complete. We set OpenBoot Prom (OBP) variables to force the domain to come to the ok prompt instead of booting an OS by setting the autoboot? property so as to demonstrate OBP commands. The “\” in the command line is an escape character, so we can enter the “?” character as a literal value. Then we bind the domain, which assigns the specified resources to the domain. This includes assigning the port used by the virtual console concentrator—5000 in the example. Finally, we start the domain, which has an effect similar to performing a “power on” operation for a real server: It loads the OBP, which then displays its ok prompt.

It’s helpful to bring up a second terminal window to watch this process. The telnet command can be issued after theldm bind command to do so. At first, no output follows the output coming from the telnet command itself (the line beginning with “Press ~?”). When ldm start ldom1 is issued, OpenBoot is loaded and outputs {0} ok.

3.4.6 Viewing a Domain
From the ok prompt, we can issue OpenBoot commands and view the domain’s devices:

This output shows that the domain is a virtual machine with its own devices—notice how the device aliases are derived from the ldm commands that defined them.

3.4.7 Installing Oracle Solaris into a Domain
Even the thrill of issuing OpenBoot commands at the ok prompt can pall after a while, so we will install Solaris 10 in the domain. Installation can be done over the network if a JumpStart or Ops Center infrastructure is available, and works just as on physical machines. For simplicity, we illustrate booting from a Solaris installation DVD image. Except for the device name alias in the boot command, this process is identical to that used when installing from a DVD on a physical machine. Notice the name of the discovered network interface: vnet0.

The rest of the Solaris installation is as usual, and is not shown here. The disk space occupied by virtual disks may be smaller than its apparent size. As demonstrated by the following lines, a 10 GB virtual disk containing a newly installed copy of Oracle Solaris takes up only a little more than 2.5 GB of disk space.

The disk space consumed will depend on how much of the virtual disk has been populated by the guest operating system and how well the data is compressed if ZFS compression is being used.

3.4.8 Observing Guest Domains from the Control Domain

From the control domain, you can use theldm command to observe the activity of the domain, and to see which CPU, memory, network disk, cryptographic accelerator, and console resources are bound to it. The following example shows the short and long forms of ldm list. The control domain’s console is SP, indicating that its console is accessed via the service processor, which is typical for Solaris on a SPARC server. The line for the guest domain shows the virtual console service port number used to access the domain’s console.

In this example, you can see the number of virtual CPUs and the physical CPUs to which they are bound. For instance, ldom1’s virtual CPU 0 is on physical CPU 4. The long listing format shows the utilization of each virtual CPU.

The STRAND column indicates the percentage of the physical CPU strand owned by that domain. In the current implementation, that value is always 100% because threads are dedicated to domains. The field is provided so scripts that process command output will continue to work if partial thread allocation is added later. The ldm list -p command option can be used to produce parsable output for easier script writing. In the following listing, ldom2 is a four-vCPU domain running at 25% average CPU utilization. The parsable format output, with or without the awk command, makes it clear that one CPU is fully used while the others are idle.

3.4.9 Viewing a Domain from the Inside
Once Oracle Solaris is installed in a domain, you can use normal commands to view its configuration. Notice that the domain has four CPUs and a network interface named vnet0 with its own MAC address. A pleasant side effect of running in a domain is that the boot process is very fast, because there is no need to perform a power-on self-test (POST) or probe physical devices.

At this point, the administrator can install application software and start using the domain.

3.4.10 Dynamic Reconfiguration
The Logical Domains technology lets the system administrator change the CPU and virtual I/O resources made available to guest domains. For example, the domain in the preceding example can be changed to use eight CPUs by issuing either of these commands: ldm add-vcpu 4 ldom2 or ldm set-vcpu 8 ldom2. Each command takes effect immediately without rebooting the guest, and the new CPU resources can be used immediately. The psrinfo command in the guest domain shows the new CPU configuration, including the date and time shown for the added CPUs.

Virtual I/O devices can also be added via the ldm add-vnet and ldm add-vdisk commands, and both I/O and CPU resources can be removed using complementary ldm rm‑vnet and ldm rm‑vdisk commands. Similarly, cryptographic accelerators can be dynamically added or removed. These commands can easily be scripted to ensure their execution at particular times or as part of a resource manager. At this writing, changing the amount of RAM in a domain requires that the domain be rebooted before the Solaris instance will recognize the change.

3.4.11 Dynamic Resource Management
The Logical Domains technology provides a policy-based resource manager that automatically adds or removes CPUs from a running domain based on its utilization and relative priority. Policies can be prioritized to ensure that important domains obtain preferential access to resources. They can also be enabled or disabled manually or based on time of day for different prime shift and off-hours policies. For example, one domain may have the highest resource needs and priority during the daytime, while a domain running batch work may operate in a more resource-intensive manner at night.

Policy rules specify the number of CPUs that a domain has, bounded by minimum and maximum values, and based on their utilization:

  • The number of CPUs is adjusted between vcpu-min and vcpu-max based on util-upper and util-lower CPU busy percentages. (All of these variables are property values associated with the policy.)
  • If CPU utilization exceeds the value of util-upper, virtual CPUs are added to the domain until the number is between vcpu-min and vcpu-max.
  • If the utilization drops below util-lower, virtual CPUs are removed from the domain until the number is between vcpu-min and vcpu-max.
  • If vcpu-min is reached, no more virtual CPUs can be dynamically removed. If vcpu-max is reached, no more virtual CPUs can be dynamically added. Manual changes to the number of CPUs can still be made using the ldm commands shown previously.
  • Multiple policies can be in effect, and are optionally controlled by tod-begin and tod-end (time of day) values.

The resource manager includes ramp-up (attack) and ramp-down (decay) controls to adjust the system’s response to workload changes, specifying the number of CPUs to add or remove based on changes in utilization, and how quickly the resource manager responds. Resource management is disabled in elastic power management mode, in which unused CPUs are powered down to reduce power consumption. The following is an example of a command creating a policy:

# ldm add-policy tod-begin=09:00 tod-end=18:00 util-lower=25 \
util-upper=75 vcpu-min=2 vcpu-max=16 attack=1 \
decay=1 priority=1 name=high-usage ldom1

This policy controls the number of CPUs for domain ldom1, is named high-usage, and is in effect between 9 a.m. and 6 p.m. The lower and upper CPU utilization settings are 25% and 75% CPU busy, respectively. The number of CPUs is adjusted between 2 and 16: One CPU is added or removed at a time (the attack and decay values). For example, if the CPU utilization exceeds 75%, a CPU is added unless ldom1 already has 16 CPUs.

The resource manager provides flexible and powerful policy-driven dynamic CPU resource management for Logical Domains that can automatically adjust CPU assignments based on CPU resource requirements and domain priorities.

3.4.12 Cloning a Domain
The Logical Domains technology makes it easy to clone systems, especially when using virtual disks residing in ZFS. A golden image instance of Oracle Solaris can be installed, patched, and customized, and then used as a master copy for multiple domains.

ZFS makes this process efficient by letting the administrator take a snapshot of a virtual disk and then create clones from it. Snapshots are read-only images of data in a ZFS file system at the time the zfs snapshot command is executed, while clones are read/write images based on the snapshot. ZFS saves disk space because only changed data takes up additional space on disk. For example, if a snapshot is made of a ZFS file system with 100 GB of data and 5 MB of data is changed, only 5 MB of additional disk space is consumed by the snapshot. Both a snapshot and the file system it is based on use common disk locations for data that is common to both. The same is true with clones: Space is consumed only for changed disk contents. At first, the disk footprint of the new domain will be negligible, as shown in the following example, but eventually it may increase if the contents of the Solaris instance diverge from the master image.

Virtual disks based in a different file system, such as UFS, can be easily replicated by using the low-tech cp command. With this approach, however, each copy of a disk will require the same disk space as the original. Given that ZFS is much more space efficient, it is the recommended technique for cloning a domain.

Before cloning a domain, be sure to shut it down to ensure that all buffers have been written and that its disk contents are stable. You may also wish to unbind the domain if its purpose is to be a template for other domains.

Then, assuming use of the same domain as in the previous example, we could issue the following commands:

Once you’ve issued the ldm commands needed to define domain ldom2

A cloned Solaris instance has the same IP address and host name as the system it was cloned from. You can choose from several methods to give it a unique identity. One strategy is to configure the guest to use DHCP at boot. Another technique is to issue the /usr/sbin/sys-unconfig command before shutting down and cloning the golden image. The first boot of the cloned image will then prompt (at the guest console) for system identification data, such as the time zone, IP address, and so on. Alternatively, you can configure the golden image domain with a unique host name and IP address that will not be given to any other domain. After booting a domain cloned from it, simply log into the clone via SSH, and change its host name and IP address via standard Solaris administrator commands. An advantage of this method is that it avoids the use of the guest console.

  Oracle VM Server for SPARC

Configuring, implementing Logical Domains in Oracle Solaris 10

Creating, viewing and installing Oracle Solaris 10 into a domain

Logical Domains migration tips for VARs in Oracle Solaris 10

This excerpt is from the book, ‘Oracle Solaris 10 System Virtualization Essentials’ by Jeff Victor, Jeff Savit, Gary Combs, Simon Hayler, Bob Netherton, published by Pearson/Prentice Hall Professional, Sept. 2010, ISBN 0-13-708188-X, Copyright 2011; for a complete list of contents, please visit the publisher site:

Dig Deeper on MSP business strategy

Cloud Computing
Data Management
Business Analytics