Miscellaneous Solaris notes

netstat -k

Determine total amount of bytes (not just packets) sent and received over a network interface:

netstat -k | less
/hme0 (or your network interface)

hme0:
ipackets 1450233797 ierrors 5183 opackets 1312917784 oerrors 0 collisions 122831
24
defer 58 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0
ifspeed 100 buff 5063 oflo 0 uflo 0 missed 107 tx_late_collisions 0
retry_error 0 first_collisions 55296 nocarrier 0 inits 246 nocanput 917395
allocbfail 0 runt 13 jabber 0 babble 0 tmd_error 0 tx_late_error 0
rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0
slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0
rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0
rx_late_collisions 0 rbytes 1773527647 obytes 3313629004 multircv 321597 multixm
t 0
brdcstrcv 328712706 brdcstxmt 162426 norcvbuf 726653 noxmtbuf 0

netstat -k provides a method to display kernel statistics recorded by the kstat facility.

prtconf usage

Display device aliases (along with other information) without being at the Open Boot Prompt ("ok" prompt):

prtconf -pv

Determine current boot-device (i.e. it is possible that the first entry in the EEPROM boot-device order is not necessarily the current boot-device):

prtconf -pv | grep bootpath

Here is a script I wrote to convert the "bootpath" into cxtxdx (logical) format. This procedure should work with Solaris 2.6 and later. If it does not work on your system, please send me the prtconf output above along with the full device path to your boot disk.

#!/bin/sh

# Determine the Solaris boot disk in cxtxdx format.

BOOTPATH=`prtconf -pv | grep bootpath | tr -d "'" | awk '{print $2}'`

if [ -n "`echo $BOOTPATH | grep "/disk"`" ] ; then
   # The bootpath contains "disk," but the /devices block device contains
   # either "sd" or "ssd"
   BOOTPATH=`echo $BOOTPATH | sed 's/disk@//'`
   BOOT_DISK=`ls -l /dev/dsk | sed -e 's/ssd@//' -e 's/sd@//' \
   | grep "$BOOTPATH" 2>/dev/null | awk '{print $9}' | sed 's/s[0-7]//'`
else
   BOOT_DISK=`ls -l /dev/dsk | grep "$BOOTPATH" 2>/dev/null | \
   awk '{print $9}' | sed 's/s[0-7]//'`
fi

if [ -n "$BOOT_DISK" ] ; then
   echo "Your boot disk is ${BOOT_DISK}."
else
   echo "Unable to determine logical boot disk."
fi

More information at sun-managers mailing list.

Determine minfree

minfree is the percentage of a file system reserved for the superuser or privileged processes. The value is 10% by default on UFS file systems.

/usr/sbin/fstyp -v raw_device | grep minfree

ex.
# /usr/sbin/fstyp -v /dev/rdsk/c0t0d0s0 | grep minfree
minfree 10% maxbpg 2048 optim time

"Blackhole"/null routing a destination IP address:

route add destination_IP 127.0.0.1 -blackhole

Note that this seems to work without the -blackhole flag, and can be used on most Unices.

All network traffic destined to destination_IP will not leave the network interface, as it is given a "next hop" of 127.0.0.1 (localhost). Note that destination_IP can still reach the host, although response traffic will not reach destination_IP, making it appear that communication is broken.

Multiple devalias entries with eeprom

ex.
# eeprom "nvramrc=devalias c0t0d0 /sbus@2,0/SUNW,socal@d,10000/sf@0,0/ssd@w220000203714f2ee,0:a devalias c3t6d0 /sbus@6,0/SUNW,socal@d,10000/sf@1,0/ssd@w2200002037093df0,0:a"

Make sure to also set use-nvramrc? to true. These device aliases will be stored after a system reset.

# eeprom "use-nvramrc?=true"

SENA disk serial numbers

Script to display disk serial numbers for fibre channel devices and devices in a Sun Enterprise Network Array (SENA):

#!/bin/sh

for disk in `format < /dev/null | egrep "[0-9]+\." | awk '{print $2}'`
do
serial=`luxadm display /dev/rdsk/${disk}s2 2>/dev/null | grep Serial | awk '{print $3}'`
echo "$disk\t$serial"
done

Example output:
c0t0d0 LAV27428
c0t2d0 LAY18677
c1t0d0 LAV00875
c1t1d0 LAY19210
c1t2d0 LAY18927
c1t3d0 LAY17116
c1t4d0 LAY19231
c1t6d0 LAY18239
c1t16d0 LAY19212
c1t17d0 LAY18981
c1t18d0 LAY19339
c1t19d0 LAY16957
c1t20d0 LAY17615
c1t22d0 LAY15851

"configuring IPv4 interfaces:ifconfig: setifflags: SIOCSLIFFLAGS: interface: Cannot assign requested address interface"

An /etc/hostname.interface file exists, but the value in /etc/hostname.interface file is not in /etc/hosts. Add the IP address and hostname listed in /etc/hostname.interface to /etc/hosts.

Determining RPM of a Sun disk

Run mkfs -m /dev/rdsk/cxtxdxs2 and multiply the rps= value (revolutions per second) by 60 to determine the RPM.

ex. # mkfs -m /dev/rdsk/c0t0d0s2
mkfs -F ufs -o nsect=133,ntrack=27,bsize=8192,fragsize=1024,cgsize=16,free=10,rps=90,nbpi=2049,opt=t,apc=0,gap=0,nrpos=8,maxcontig=16 /dev/rdsk/c0t0d0s2 412964

90 (rps) * 60 (seconds) = 5400 RPM.

"UX: groupadd: ERROR: group is already in use. Choose another."

If you attempt to add a group with a name longer than 8 characters, you may see this error message with Solaris even if group is not in /etc/group or the NIS group map. You may manually add groups longer than 8 characters to /etc/group to overcome this limitation, although grpck will complain about these entries.

Solaris routing with multiple interfaces

In this example, we have an hme2 network interface with an IP address of 65.201.213.117, a network address of 65.201.213.112, and a subnet mask of 255.255.255.240. The interface was initially enabled using the default subnet mask and broadcast address.

ifconfig hme2
hme2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 65.201.213.117 netmask ff000000 broadcast 65.255.255.255

Routing Table: IPv4
Destination           Gateway           Flags Ref   Use   Interface
-------------------- -------------------- ----- ----- ------ ---------
65.0.0.0             65.201.213.117        U        1      8 hme2

To correct the routing table:

1. Specify the correct subnet mask and broadcast address with ifconfig.
ifconfig 65.201.213.117 netmask 255.255.255.240 broadcast 65.201.213.127

2. Add an /etc/netmasks entry for the correct netmask to be used after system reboot.
65.201.213.112    255.255.255.240

Corrected routing table:

Routing Table: IPv4
Destination           Gateway           Flags Ref   Use   Interface
-------------------- -------------------- ----- ----- ------ ---------
65.201.213.112       65.201.213.117        U        1      1 hme2

"/usr/share/man/windex: No such file or directory"

Run catman -w as root to rebuild the "windex" database that is used by the whatis and man -f and -k options.

List installed packages by size

The following script will list all installed Solaris packages with an estimate of their size using the ouput of pkginfo. A better implementation would be a Perl script that would look at the /var/sadm/install/contents file.

#!/bin/sh

printf "%-10s %-10s %-s\n" "Name" "Size (k)" "Description"
printf "%-10s %-10s %-s\n" "----" "--------" "-----------"

pkginfo | while read line
do
   set $line
   category=$1
   name=$2
   description=`pkginfo -l $name | grep DESC: | awk -F: '{print $2}'`
   size=`pkginfo -l $name | grep "blocks used" | awk '{print $1}'`
   [ -n "$size" ] && size_in_kbytes=`echo "$size * 512 / 1024" | bc`
   printf "%-10s %-8d %-s\n" "$name" "${size_in_kbytes}" "$description" 2>/dev/null
done

A "block" (disk block) referred to in pkginfo -l output or df output is 512 bytes. By default, Solaris UFS file systems are created using 8K blocks (filesystem blocks).

"Clock board TOD does not match TOD on any IO board."

Run the command copy-clock-tod-to-io-boards from the Open Boot Prompt ("ok" prompt).

patchadd return codes

The "Return codes" listed when installing the Sun Recommended Patch Clusters are listed toward the top of the patchadd script. Here are some common return codes:

#               2       Attempt to apply a patch that's already been applied
#               8       Attempting to patch a package that is not installed
#               25      A required patch is not applied

Mounting an ISO image

The following instructions are for Solaris 8 and later. You must be root to mount the ISO image.

lofiadm -a absolute_path_to_ISO
mount -F hsfs -r /dev/lofi/1 mount_point

ex. mounting /home/hutch/cdrom.iso to /tmp/mnt
lofiadm -a /home/hutch/cdrom.iso
/dev/lofi/1

mount -F hsfs -r /dev/lofi/1 /tmp/mnt

"WARNING: Timed out waiting for NIS to come up"

If a Solaris system is configured as a NIS client and there is a problem reaching a NIS server, the system will hang indefinitely when booting. To fix,

1. Perform a send brk to get to the Open Boot Prompt ("ok" prompt).
2. Boot to single-user mode with boot -s
3. Temporarily disable RPC services or remove /etc/defaultdomain to prevent NIS from starting at system boot.
mv /etc/rc2.d/S71rpc /etc/rc2.d/NOS71rpc
-- or --
rm /etc/defaultdomain
4. Type exit to continue booting your system to your default runlevel.

The system will now boot. Begin troubleshooting problems reaching the NIS server with /etc/init.d/rpc start (if you disabled RPC) or domainname your_NIS_domainname;/usr/lib/netsvc/yp/ypstart (if you removed /etc/defaultdomain).

Once you determine the problem, re-enable RPC services with mv /etc/rc2.d/NOS71rpc /etc/rc2.d/S71rpc or re-create /etc/defaultdomain, depending on which method of disabling NIS you chose.

"Blackhole"/null routing a destination IP address

route add destination_IP 127.0.0.1 -blackhole

Note that this seems to work without the -blackhole flag, and can be used on most Unices.

All network traffic destined to destination_IP will not leave the network interface, as it is given a "next hop" of 127.0.0.1 (localhost). Note that destination_IP can still reach the host performing the null routing, although response traffic will not reach destination_IP, making it appear that communication is broken.

"umount: I/O error"
umount: cannot unmount /mount_point

I received this error on a Solaris 8 system when the disk containing a Veritas volume failed. After killing all the processed identified with fuser -c /mount_point and confirming no other processes were in use in the file system with lsof +D /mount_point, I had to issue the umount -f /mount_point command to forcibly unmount the file system:

# umount /file_system
umount: I/O error
umount: cannot unmount /file_system
# fuser -c /file_system
/file_system:
# lsof +D /file_system
# umount -f /file_system
# mount | grep /file_system
#

For Solaris users, only Solaris 8 and later provide the umount -f ("force") flag. Other Unices may have the "force" flag for umount.

Volume Management (vold) problems

Starting vold via /etc/init.d/volmgt start or /usr/sbin/vold will occasionally fail.

# /usr/sbin/vold
Mon Dec 22 14:49:07 2003 fatal: mounting of "/vol" failed

# cd /vol
NFS server for volume management (/vol) not responding still trying
NFS getattr failed for server for volume management (/vol): error 23 (RPC: Unitdata error)
/vol: bad directory

To fix the problem, unmount the /vol directory (you may have to use the -f "force" flag in umount for Solaris 8 and later) and restart volume management.

"No directory! Logging in with home=/"

This error message may occur on Solaris systems when the automount daemon (automountd) is already running when NIS is started (/usr/lib/netsvc/yp/ypstart). To fix this, restart the automount daemon via /etc/init.d/autofs stop;/etc/init.d/autofs start and re-login.

Sun Explorer

The Sun Explorer Data Collector is a tool that gathers extensive system information primarly for the use of Sun engineers. The latest version of the tool may be downloaded at http://sunsolve.sun.com/diag/explorer/SUNWexplo.tar.Z

Problems starting vold

If the volume management service will not start via /etc/init.d/volmgt start or /usr/sbin/vold, and the following errors are listed in /var/adm/messages and /var/adm/vold.log, you may need to unmount /vol. Run fuser -c /vol to see which process ID(s) are using /vol. Kill these process(es), and restart volume management.

/var/adm/messages errors:
/usr/sbin/vold[9808]: svc_tli_create: could not bind to requested address: No Error
/usr/sbin/vold[9808]: svc_tli_create: Cannot create server handle

/var/adm/vold.log errors:
fatal: svc_tli_create: Cannot create server handle

Booting off an external CD-ROM

To boot off an external CD-ROM, determine the device path with probe-scsi-all from the Open Boot PROM (OBP or "ok" prompt). Note: you may want to temporarily perform a setenv auto-boot? false and reset-all if OBP warns you that executing probe-scsi-all could hang your system.

Relevant probe-scsi-all output:

/sbus@b,0/SUNW,fas@3,8800000
Target 3
Unit 0 Removable Read Only device PLEXTOR CD-R PX-R820T 1.0811/21/00
16:00

In this example, the SCSI target ID ("Target") is 3, and the physical path to the external CD-ROM is /sbus@b,0/SUNW,fas@3,8800000. To boot a Solaris Operating Environment CD using this external CD-ROM, you would issue the command:

{e} ok boot /sbus@b,0/SUNW,fas@3,8800000/sd@3,0:f

In the command above, sd@3,0 refers to the SCSI target (3) and LUN/Unit (0).

Note: when connecting an external CD-ROM device, make sure that the CD-ROM drive and system are both powered off. Connect the CD-ROM, and power on the CD-ROM and then the system. Failure to follow this procedure may result in the following errors:

{e} ok boot /sbus@b,0/SUNW,fas@3,8800000/sd@3,0:f
Boot device: /sbus@b,0/SUNW,fas@3,8800000/sd@3,0:f File and args:
Bad magic number in disk label
Can't open disk label package

Can't open boot device

Repairing a hard disk with bad sectors

The format command may be used to repair a hard disk that is experiencing read/write errors.

Example error:

WARNING: /sbus@3,0/SUNW,fas@3,8800000/sd@0,0 (sd120):
        Error for Command: read                    Error Level: Retryable
        Requested Block: 577712                    Error Block: 577796
        Vendor: SEAGATE                            Serial Number: 9737K75183
        Sense Key: Media Error
        ASC: 0x16 (data sync mark error), ASCQ: 0x0, FRU: 0xd2

While the format command allows you to repair an individual block, it is probably best to run analyze and perform at least a read test, as there may be more than one bad block on the disk.

The analyze command will automatically repair errors it finds when scanning the hard disk for bad blocks.

analyze> read
Ready to analyze (won't harm SunOS). This takes a long time,
but is interruptable with CTRL-C. Continue? y

        pass 0
WARNING: /sbus@3,0/SUNW,fas@3,8800000/sd@0,0 (sd120):
        Error for Command: read                    Error Level: Retryable
        Requested Block: 577800                    Error Block: 577800
        Vendor: SEAGATE                            Serial Number: 9737K75183
        Sense Key: Media Error
        ASC: 0x16 (data sync mark error), ASCQ: 0x0, FRU: 0xd2
Medium error during read: block 577800 (0x8d108) (267/8/0)
ASC: 0x16   ASCQ: 0x0
Repairing hard error on 577800 (267/8/0)...ok.

Back to brandonhutchinson.com.