-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

===========================================================================
             AUSCERT External Security Bulletin Redistribution

                        ESB-2008.0305 -- [Solaris]
Solaris Systems Running VERITAS (VxVM) 4.1/5.0 With Certain HBAs Installed
           Are Unable to Complete I/O Operations and May Become
                     Unresponsive With Certain Targets
                               25 March 2008

===========================================================================

        AusCERT Security Bulletin Summary
        ---------------------------------

Product:              VERITAS Storage Foundation 4.1 Software
                      VERITAS Volume Manager 4.1 Software
                      VERITAS Storage Foundation 5.0 Software
Publisher:            Sun Microsystems
Operating System:     Solaris
Impact:               Denial of Service
Access:               Remote/Unauthenticated

Original Bulletin:    
  http://sunsolve.sun.com/search/printfriendly.do?assetkey=1-66-200561-1

- --------------------------BEGIN INCLUDED TEXT--------------------

Solution Type: Sun Alert
   Solution  200561 :   Solaris Systems Running VERITAS (VxVM) 4.1/5.0
   With Certain HBAs Installed Are Unable to Complete I/O Operations and
   May Become Unresponsive With Certain Targets          
   Previously Published As: 103107

   Bug ID: 6607251, 6472115

   Product
   VERITAS Storage Foundation 4.1 Software
   VERITAS Volume Manager 4.1 Software
   VERITAS Storage Foundation 5.0 Software

   Date of Workaround Release: 02-NOV-2007

   Date of Resolved Release: 13-Feb-2008

   SA Document Body
   Solaris Systems Running VERITAS (VxVM) 4.1/5.0 With Certain HBAs 
   Installed Are Unable to Complete I/O Operations and May Become 
   Unresponsive With Certain Targets

   1. Impact

   Solaris 8, 9 or 10 systems running VERITAS (VxVM) 4.1/5.0 with certain
   HBAs installed (with Qlogic kernel driver code & Qlogic FW) are unable
   to complete I/O operations and may become unresponsive with certain
   targets.
   2. Contributing Factors

   This issue can occur in the following releases:
     * VERITAS VxVM versions 4.1mp1 (for Solaris 8, 9 and 10) with
       limited release patch 122059-02
     * VERITAS VxVM versions 4.1mp2 (for Solaris 8, 9 and 10) with patch
       117080-07 (VRTSvxvm 4.1MP2 Maintenance Patch)
     * VERITAS VxVM version 5.0 (for Solaris 8, 9 and 10)
     * VERITAS VxVM versions 5.0mp1 (for Solaris 8, 9 and 10) with patch
       122058-09 (VRTSvxvm 5.0MP1 Maintenance Patch)
     * VERITAS VxVM versions 5.0mp1 (for Solaris 8, 9 and 10) with patch
       124361-04 (VRTSvxvm 5.0_MP1_RP3 Rolling Patch 3)
     * VERITAS VxVM version 5.0. x64 (for Solaris 10 x64)

   with any of the following Sun-QLogic (2Gb/4Gb) HBAs installed:
     * StorEdge 2Gb FC PCI Single Channel Network Adapter
     * Sun StorageTek PCI-X 2Gb FC Single Port HBA
     * StorEdge 2Gb FC PCI Dual Channel Network Adapter
     * Sun StorageTek PCI-X 4Gb FC Single & Dual Port HBA
     * Sun StorageTek PCI Express 4Gb FC Single & Dual Port HBA
     * Sun StorageTek 4Gb FC PCI-E ExpressModule HBA
     * Any QLogic (2Gb/4Gb) HBA using the Sun qlc driver module

   Notes:
    1. The underlying issue is not one of VxVM,  but the qlc Qlogic
       kernel driver code & Qlogic FW.
    2. This issue will only been see if their are FC errors on the port
       that's running the qlc driver code

   To determine if these cards and/or drivers are installed on a system,
   the modinfo(1M) command can be run as in the following examples:
    $ modinfo|grep vx
    28  124febf  34f93 251   1  vxdmp (VxVM 4.1_MP2_RP1.2: DMP Driver)
    30 78216000 21cb09 252   1  vxio (VxVM 4.1_MP2_RP1.2 I/O driver)
    32  128144d   14a9 253   1  vxspec (VxVM 4.1_MP2_RP1.2 control/stat)
    282 785a930f    cab 248   1  vxportal (VxFS 4.1_REV-MP2b portal driver)
    283 787f2000 17100d   8   1  vxfs (VxFS 4.1_REV-MP2b SunOS 5.9)

    $ modinfo|grep qlc
    50  12d421d  cd574 153   1  qlc (SunFC Qlogic FCA v20070212-2.19)

   3.  Symptoms

   System and IO hangs are coincident with following errors in messages:
    WARNING: qlc(0): isr, Internal Parity/Pause Error - hccr=8000h,
       stat=40008110h, count=9388
    WARNING: qlc(0): isr, Internal Parity/Pause Error - hccr=0h,
       stat=40008110h, count=9389

   Or:
    WARNING: /pci@3,700000/SUNW,qlc@0,1/fp@0,0/ssd@w50060e800327572c,5a
    (ssd1381):
    undecodable sense information: 0x0 0x0 0x0 0x1 0x0 0x0 0x0 0x2 0xff
    0xff 0xff 0xff 0x13 0x7c 0x70 0xc0 0x0 0x0 0x0 0x1-(assumed fatal)

   Analysis indicates the parity errors are the result of untagged scsi
   CDBs sent by vxdmp versions 4.1mp2, and later, during error recovery.

   Sequence of events leading to system hang are as follows:

   2G HBA
    1. error condition on fiber requires vxdmp retry
    2. vxdmp resends IO without a tagged CDB
    3. HBA responds with error
    4. path is failed - possibly all paths are failed
    5. vxconfigd appears to hang due to being overwhelmed processing
       errors

   4G HBA
    1. error condition on fiber requires vxdmp retry
    2. vxdmp resends io without a tagged CDB
    3. HBA responds wtih error
    4. HBA enters an infinite error report loop
    5. no further IO processing takes place (all subsequent IO hangs)

   4. Workaround

   Symantec has provided a tech tip at:
   http://seer.entsupport.symantec.com/docs/292445.htm

   Customers running 5.0 with Qlogic 2G or 4G HBAs are encouraged to
   install 5.0mp1 and set dmp_fast_recovery=off, as in the following
   example:
    root# vxdmpadm gettune all |grep fast_recovery
    dmp_fast_recovery              on               on
    root# vxdmpadm settune dmp_fast_recovery=off
    Tunable value will be changed immediately
    root# vxdmpadm gettune all |grep fast_recovery
    dmp_fast_recovery             off               on
    root# cat /etc/vx/dmppolicy.info
    arraytype
    #
    arrayname
    #
    enclosure
    #
    Tunables
    dmp_fast_recovery=off
    #
    root#

   Customers running 5.0mp1 with Qlogic 2G or 4G HBAS are encouraged to
   set the following parameters:
    root# vxdmpadm gettune all |grep fast_recovery
    dmp_fast_recovery              on               on
    root# vxdmpadm settune dmp_fast_recovery=off
    Tunable value will be changed immediately
    root# vxdmpadm gettune all |grep fast_recovery
    dmp_fast_recovery             off               on
    root# cat /etc/vx/dmppolicy.info
    arraytype
    #
    arrayname
    #
    enclosure
    #
    Tunables
    dmp_fast_recovery=off
    #
    root#

   5. Resolution

   This issue is addressed in the following (including new qlc driver
   patches which address the underlying fix for the hangs, unable to
   complete I/O, and qlc errors):

   SPARC Platform
     * Solaris 10 QLC driver patch 125166-07 or later
     * Solaris 9 SAN 4.4.x: QLC driver patch 113042-19 or later
     * VERITAS VxVM 4.1MP2_RP3 patch 124358-05 or later (for Solaris 8, 9
       and 10)
     * VERITAS VxVM  5.0MP1_RP4 124361-05 or later (for Solaris 8, 9, and
       10)

   x86 Platform
     * Solaris 10 QLC driver patch 125165-07 or later
     * VERITAS VxVM 5.0 x64_RP1 patch 128060-02 or later (for Solaris 10
       x64)

   Symantec has provided a tech tip at:
   http://seer.entsupport.symantec.com/docs/292445.htm

   This Sun Alert notification is being provided to you on an "AS IS"
   basis. This Sun Alert notification may contain information provided by
   third parties. The issues described in this Sun Alert notification may
   or may not impact your system(s). Sun makes no representations,
   warranties, or guarantees as to the information contained herein. ANY
   AND ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION
   WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR
   NON-INFRINGEMENT, ARE HEREBY DISCLAIMED. BY ACCESSING THIS DOCUMENT
   YOU ACKNOWLEDGE THAT SUN SHALL IN NO EVENT BE LIABLE FOR ANY DIRECT,
   INDIRECT, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES THAT ARISE
   OUT OF YOUR USE OR FAILURE TO USE THE INFORMATION CONTAINED HEREIN.
   This Sun Alert notification contains Sun proprietary and confidential
   information. It is being provided to you pursuant to the provisions of
   your agreement to purchase services from Sun, or, if you do not have
   such an agreement, the Sun.com Terms of Use. This Sun Alert
   notification may only be used for the purposes contemplated by these
   agreements.
   Copyright 2000-2008 Sun Microsystems, Inc., 4150 Network Circle, Santa
   Clara, CA 95054 U.S.A. All rights reserved

- --------------------------END INCLUDED TEXT--------------------

You have received this e-mail bulletin as a result of your organisation's
registration with AusCERT. The mailing list you are subscribed to is
maintained within your organisation, so if you do not wish to continue
receiving these bulletins you should contact your local IT manager. If
you do not know who that is, please send an email to auscert@auscert.org.au
and we will forward your request to the appropriate person.

NOTE: Third Party Rights
This security bulletin is provided as a service to AusCERT's members.  As
AusCERT did not write the document quoted above, AusCERT has had no control
over its content. The decision to follow or act on information or advice
contained in this security bulletin is the responsibility of each user or
organisation, and should be considered in accordance with your organisation's
site policies and procedures. AusCERT takes no responsibility for consequences
which may arise from following or acting on information or advice contained in
this security bulletin.

NOTE: This is only the original release of the security bulletin.  It may
not be updated when updates to the original are made.  If downloading at
a later date, it is recommended that the bulletin is retrieved directly
from the author's website to ensure that the information is still current.

Contact information for the authors of the original document is included
in the Security Bulletin above.  If you have any questions or need further
information, please contact them directly.

Previous advisories and external security bulletins can be retrieved from:

        http://www.auscert.org.au/render.html?cid=1980

If you believe that your computer system has been compromised or attacked in 
any way, we encourage you to let us know by completing the secure National IT 
Incident Reporting Form at:

        http://www.auscert.org.au/render.html?it=3192

===========================================================================
Australian Computer Emergency Response Team
The University of Queensland
Brisbane
Qld 4072

Internet Email: auscert@auscert.org.au
Facsimile:      (07) 3365 7031
Telephone:      (07) 3365 4417 (International: +61 7 3365 4417)
                AusCERT personnel answer during Queensland business hours
                which are GMT+10:00 (AEST).
                On call after hours for member emergencies only.
===========================================================================

-----BEGIN PGP SIGNATURE-----
Comment: http://www.auscert.org.au/render.html?it=1967

iQCVAwUBR+hbByh9+71yA2DNAQKuqwP/VrKfCkAMggN9JFvQ/s6o8FUWy1cdJYcR
LdNf+cnbqjwBtD7iGJhuk1Um6bid3U3hrSTNGucoIf0u3ENDpj0ubTU5wtZa3TY5
4xplfUhHl4OYeTBSoXjkhrMNOYyP42pSynr1+3DXfRGyGENTHE534KhMU0/Q6HzP
1RYdtj/xHiY=
=yvas
-----END PGP SIGNATURE-----