Operating System:

[Solaris]

Published:

15 February 2007

Protect yourself against future threats.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

===========================================================================
             AUSCERT External Security Bulletin Redistribution

                        ESB-2007.0039 -- [Solaris]
          Solaris 10 System Panics May be Seen Under High TCP/IP
         Stress Due to Race Conditions While Closing TCP Instances
                             15 February 2007

===========================================================================

        AusCERT Security Bulletin Summary
        ---------------------------------

Product:              tcp_close
Publisher:            Sun Microsystems
Operating System:     Solaris 10
Impact:               Denial of Service
Access:               Remote/Unauthenticated

Original Bulletin:    
  http://sunsolve.sun.com/search/printfriendly.do?assetkey=1-26-102551-1

Revision History:  February 15 2007: Various updates including changes to 
                                     the resolution and the 
                                     relief/workaround sections.
                   January 22 2007: Initial Release

- --------------------------BEGIN INCLUDED TEXT--------------------

Sun(sm) Alert Notification
     * Sun Alert ID: 102551
     * Synopsis: Solaris 10 System Panics May be Seen Under High TCP/IP
       Stress Due to Race Conditions While Closing TCP Instances
     * Category: Availability
     * Product: Solaris 10 Operating System
     * BugIDs: 6404207
     * Avoidance: Patch
     * State: Resolved
     * Date Released: 09-Aug-2006, 13-Feb-2007
     * Date Closed: 13-Feb-2007
     * Date Modified: 19-Jan-2007, 13-Feb-2007

1. Impact

   Systems may experience a race condition in the tcp_close() function
   which will result in a system panic.

   The following Sun Alerts describe TCP/IP issues for the new SPARC
   systems which use CMT processors.
     * Sun Alert ID: 102551
     * Sun Alert ID: 102553
     * Sun Alert ID: 102554

2. Contributing Factors

   This issue can occur in the following releases:

   SPARC Platform
     * Solaris 10 without patch 119998-01

   x86 Platform
     * Solaris 10 without patch 119999-02

   Notes: Solaris 8 and 9 are not impacted by this issue.

   Extremely heavy TCP/IP loads with many tens of thousands of rapidly
   opening/closing TCP connections.

   CMT processors (Chip Multi-Threading technology) are more likely to
   sustain the throughput required, while still being able to context
   switch rapidly enough to reproduce this. It is highly unlikely that
   this issue will be seen on other systems.

   There are two classes of system using CMT processors. These can be
   identified using the following commands:

   a) Systems with Sun4v kernel architecture use CMT processors.

   These are identified by the command:
     % uname -m

   which will return "sun4v".

   b) UltraSPARC IV and UltraSPARC IV+ systems use CMT processors.

   These are identified by the command:
     % prtconf -pv | grep "SUNW,UltraSPARC-IV"

   which will return a non-NULL result something similar to: compatible:
   'SUNW,UltraSPARC-IV'.

3. Symptoms

   Several symptoms are seen when threads race to close/free a TCP eager.

   a) A panic string and stacktrace similar to the following:
   BAD TRAP: type=31 rp=2a104df53f0 addr=0 mmu_fsr=0 occurred
   in module "unix" due to a NULL pointer dereference

   atomic_add_int_nv+0x0(0x0, 0xffffffffffffffff)
   crfree+0xc(0x0, , 0x0, 0x3000aad3a88, 0x3000134e000)
   tcp_close+0x144(0x6002df0cb48, 0x3, 0x600362c2318)
   qdetach+0x90(0x6002df0cb48, 0x1, 0x3, 0x600362c2318, 0x0)
   strclose+0x3c0(0x30015add440, 0x3, 0x600362c2318)
   socktpi_close+0x158(0x30015add440, 0x3, 0x1, 0x0, 0x600362c2318)
   fop_close+0x20(0x30015add440, 0x3, 0x1, 0x0, 0x600362c2318)
   closef+0x4c(0x60013c19f80)
   closeandsetf+0x37c(0x7, 0x0)
   close+0x8(, 0xff233244, 0xc6000, 0xd2fc8, 0x5449fd11, 0x0)
   syscall_trap32+0xcc()


   b) A panic string and one of the two following stacktraces:
   panic string:   CONN_DEC_REF: connp(60024197840) has ref = 0

        ------
   panicsys+0x48(0x13b37d8, 0x2a100ba19d0, 0x1844e20, 0x1, ...
   vpanic_common+0x78(0x13b37d8, 0x2a100ba19d0, ...
   cmn_err+0x98(0x3, 0x13b37d8, 0x60024197840, ...
   squeue_drain+0x2e0(0x60001935480, 0x2, 0x3a7a, , , 0x3a79)
   squeue_worker+0x88(0x60001935480, 0x0)
   thread_start+0x4()
        ------

        ------
   panicsys+0x48(0x1076058, 0x2a1113775c8, 0x1844c20, ...
   vpanic_common+0x78(0x1076058, 0x2a1113775c8, ...
   panic+0x1c(0x1076058, 0x181c430, 0x3002d9ad1e0, ...
   mutex_enter(0x3002d9ad1e0) ...
   tcp_close+0x84(0x313c64a4ff8, ...
   qdetach+0x90(0x313c64a4ff8, 0x1, ...
   strclose+0x3c0(0x3005c8ab1c0, 0x3, ...
   socktpi_close+0xe4(0x3005c8ab1c0, ...
   fop_close+0x20(0x3005c8ab1c0, 0x3, ...
   closef+0x4c(0x313bfc555e0)
   closeandsetf+0x37c(0x7, 0x0)
   close+0x8(, 0xff233244, 0xc7400...
   syscall_trap32+0xcc()

   ------ 

4. Relief/Workaround

   There is no workaround. Please see Resolution section below.

5. Resolution

   This issue is addressed in the following releases:

   SPARC Platform
     * Solaris 10 with patch 119998-01 or later

   x86 Platform
     * Solaris 10 with patch 119999-02 or later

Change History

   19-Jan-2007:
     * Updated Relief/Workaround section with Temporary patches

   13-Feb-2007:
     * Removed CR 102551 which is addressed in Sun Alert 102796
     * Updated Contributing Factors, Relief/Workaround and Resolutions
       sections
     * State: Resolved

   This Sun Alert notification is being provided to you on an "AS IS"
   basis. This Sun Alert notification may contain information provided by
   third parties. The issues described in this Sun Alert notification may
   or may not impact your system(s). Sun makes no representations,
   warranties, or guarantees as to the information contained herein. ANY
   AND ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION
   WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR
   NON-INFRINGEMENT, ARE HEREBY DISCLAIMED. BY ACCESSING THIS DOCUMENT
   YOU ACKNOWLEDGE THAT SUN SHALL IN NO EVENT BE LIABLE FOR ANY DIRECT,
   INDIRECT, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES THAT ARISE
   OUT OF YOUR USE OR FAILURE TO USE THE INFORMATION CONTAINED HEREIN.
   This Sun Alert notification contains Sun proprietary and confidential
   information. It is being provided to you pursuant to the provisions of
   your agreement to purchase services from Sun, or, if you do not have
   such an agreement, the Sun.com Terms of Use. This Sun Alert
   notification may only be used for the purposes contemplated by these
   agreements.

   Copyright 2000-2006 Sun Microsystems, Inc., 4150 Network Circle, Santa
   Clara, CA 95054 U.S.A. All rights reserved

- --------------------------END INCLUDED TEXT--------------------

You have received this e-mail bulletin as a result of your organisation's
registration with AusCERT. The mailing list you are subscribed to is
maintained within your organisation, so if you do not wish to continue
receiving these bulletins you should contact your local IT manager. If
you do not know who that is, please send an email to auscert@auscert.org.au
and we will forward your request to the appropriate person.

NOTE: Third Party Rights
This security bulletin is provided as a service to AusCERT's members.  As
AusCERT did not write the document quoted above, AusCERT has had no control
over its content. The decision to follow or act on information or advice
contained in this security bulletin is the responsibility of each user or
organisation, and should be considered in accordance with your organisation's
site policies and procedures. AusCERT takes no responsibility for consequences
which may arise from following or acting on information or advice contained in
this security bulletin.

NOTE: This is only the original release of the security bulletin.  It may
not be updated when updates to the original are made.  If downloading at
a later date, it is recommended that the bulletin is retrieved directly
from the author's website to ensure that the information is still current.

Contact information for the authors of the original document is included
in the Security Bulletin above.  If you have any questions or need further
information, please contact them directly.

Previous advisories and external security bulletins can be retrieved from:

        http://www.auscert.org.au/render.html?cid=1980

If you believe that your computer system has been compromised or attacked in 
any way, we encourage you to let us know by completing the secure National IT 
Incident Reporting Form at:

        http://www.auscert.org.au/render.html?it=3192

===========================================================================
Australian Computer Emergency Response Team
The University of Queensland
Brisbane
Qld 4072

Internet Email: auscert@auscert.org.au
Facsimile:      (07) 3365 7031
Telephone:      (07) 3365 4417 (International: +61 7 3365 4417)
                AusCERT personnel answer during Queensland business hours
                which are GMT+10:00 (AEST).
                On call after hours for member emergencies only.
===========================================================================

-----BEGIN PGP SIGNATURE-----
Comment: http://www.auscert.org.au/render.html?it=1967

iQCVAwUBRdOoOCh9+71yA2DNAQKeRwP/RnbATt8w/gRkI0VgpOsncwFv580ZMM5h
FZywEl8rR3h8/i9gOEDgNsVVyTH7du3UToEcdT7RZKlDjVeJBoTnIyGdl20fMRBJ
Qym+X3l6iXrBokmqOY02nusH6XRGSvuZD0ovtbKgCzToNdzYttjPhVXmIU7YhxBf
Sg6ebD8yfVY=
=/u42
-----END PGP SIGNATURE-----