Generating a memory dump for an IIS 6 hang

This post follows up on my sample demonstration and explains how to get a memory dump from a production web server crash.

Environment:
ASP .NET 3.5
IIS 6
Windows Server 2003 64bit
Windbg 6.11.0001.404 AMD64

w3wp was crashing on the server with this in the event log.

Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 1000
Date: 3/3/2010
Time: 1:22:18 PM
User: N/A
Computer: _____________
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d691cc, faulting module kernel32.dll, version 5.2.3790.4480, stamp 49c51cdd, debug? 0, fault address 0x0000000000027ded.


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

I set adplus to get a memory dump on hang.

After spelunking around I found out I was debugging the wrong dump file w3wp.exe-APPNAME

decided to debug a different dump file: w3wp.exe_-CRMAppPool
.sympath SRV*c:\debugsymbols*http://msdl.microsoft.com/download/symbol
.loadby sos mscorwks
!threads

ThreadCount: 11
UnstartedThread: 0
BackgroundThread: 11
PendingThread: 0
DeadThread: 0
Hosted Runtime: yes
                                              PreEmptive                                                Lock
       ID OSID        ThreadOBJ     State   GC     GC Alloc Context                  Domain           Count APT Exception
XXXX    1 1a14 00000000001aa520   1808220 Disabled 0000000000000000:0000000000000000 00000000055650d0     1 Ukn (Threadpool Worker) System.StackOverflowException (00000000100b10d0)
XXXX    2 2014 000000000019d5c0      b220 Enabled  0000000000000000:0000000000000000 0000000000122e20     0 Ukn (Finalizer)
XXXX    3 2298 0000000005722570      1220 Enabled  0000000000000000:0000000000000000 0000000000122e20     0 Ukn
XXXX    4 1524 000000000556b570    80a220 Enabled  0000000000000000:0000000000000000 0000000000122e20     0 Ukn (Threadpool Completion Port)
XXXX    5  904 000000000556bb40   200b220 Enabled  0000000000000000:0000000000000000 00000000055650d0     0 Ukn
XXXX    7 2228 000000000c7e17b0   880a220 Enabled  0000000000000000:0000000000000000 0000000000122e20     0 Ukn (Threadpool Completion Port)
XXXX    8 2324 000000000c6ee560   200b220 Enabled  0000000000000000:0000000000000000 00000000055650d0     0 Ukn
XXXX    9 19f8 00000000055c7480   200b220 Enabled  0000000000000000:0000000000000000 00000000055650d0     0 Ukn
XXXX    a 2104 0000000005b8b9e0   180b220 Enabled  0000000000000000:0000000000000000 00000000055650d0     1 Ukn (Threadpool Worker)
XXXX    b 213c 000000000d039d10   180b220 Enabled  0000000000000000:0000000000000000 0000000000122e20     0 Ukn (Threadpool Worker)
XXXX    6 2370 000000000de00b90   200b220 Enabled  0000000000000000:0000000000000000 00000000055650d0     1 Ukn

Aha!  three threads with locks and one with a StackOverflowException.

The column with the XXXX is supposed to list out the native thread so I can switch to it and see the problem.

0:000> ~

.  0  Id: 226c.2274 Suspend: -1 Teb: 000007ff`fffa2000 Unfrozen

When this memory dump was taken the native thread had already been cleaned up.

A KB article provided instructions on running an action when a process is orphaned.

Following those instructions I get a log file but no .dmp :(

I modify action.cmd changing %COMMAND% to %COMMAND% >> %LOG%

Now the log displays: 

Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

Cannot debug pid 8220, Win32 error 0n87

    "The parameter is incorrect."

Debuggee initialization failed, Win32 error 0n87

    "The parameter is incorrect."

Hmmm... maybe I can manually take a dump.

Using iisapp I see:

W3WP.exe PID: 11072   AppPoolId: ClientProject
W3WP.exe PID: 10824   AppPoolId: 3rdPartyVendor
W3WP.exe PID: 4836   AppPoolId: AnotherClientProject
W3WP.exe PID: 8220   AppPoolId: CRMAppPool

create problem.  wait 30 seconds:

C:\>"C:\Program Files\Debugging Tools for Windows (x64)\cdb.exe" -c ".dump /o /m
a c:\crash_PID_11072_3_03_2010_10_01_16_54.dmp;q" -p 11072

success!  Using this dump we were able to identify the problem.

Becoming a Better Listener

Today I received my first downvote on StackOverflow (a programmer question and answer site).  This means the downvoter said my answer sucked.  Fortunately they left a comment explaining why.  Immediately I thought "Hey! I'm right!" My answer is technically correct.  I think?.  I read the other two provided responses and noticed they both referenced a 50x50 Green square.  Mine did not.  Then I read the question again.  Oh, they want a green 50x50 square.  I didn't realize that!

I answered the question in as little time as possible without making a good attempt at understanding what they were trying to do.  I don't even know if my code really solves their problem because I copy and pasted code from my blog and put a warning on my answer "this worked two and a half years ago."  I told myself it was OK though because I put a smiley face at the end of the sentence to feel better about submitting an untested answer.

What happened after that?  Two people took time out of their day to improve the formatting and clarity of my answer and then a third person took time out to tell me why my answer wasn't good.  My answer did not provide value to the community.  Better answerers came along and posted more concise answers that directly solved the problem.  I had wasted community member's time.  I deleted my answer.

In the real world I do the same thing.  When people talk often times I have a response ready before they are done talking.  When someone criticizes me I often interrupt stating why I am right.  Listening is key to communication and learning.  I am working on being a better listener, and if you catch me not listening, help me out :)

Geeking out in Lansing

In 2008 I lived in Detroit and struggled to find like minded individuals to hang out with.  Individuals who code for fun and want to fix problems in their communities.  I ended up spending quite a few nights in Ann Arbor which has a similar tech scene to Lansing but a bit larger.  I enjoyed new events like CoffeeHouseCoders, Ann Arbor New Tech, and the TechBrewery.

I'm happy to report Lansing now has similar events so I can get all the goodness I was getting in Southeast MI right here at home.

The inaugural Lansing CoffeeHouseCoders meeting is 1/19/2011.  If you like to code in your free time in a social atmosphere, this is the place for you!

Hackers & Hustlers is a group focused on startup culture.  Attendees will hear entrepreneurs who are actually doing it give a talk or a pitch on their product.  The first meeting is 1/26.

In 2010 Lansing also got a coworking space, Second Gear.  More and more businesses are enabling their work force to work whenever and wherever they want.  For me often times that means I need to be somewhere for an evening event and I spend the day in a coffee shop.  That works OK, but I like to work near other programmers so I can learn from them and bounce ideas off of them.  Second Gear is meeting that need for the Lansing community.

Storing User Settings

I mentioned last week I want to store all settings for my application as user settings so I won't write to app.config and require administrative privileges.  I didn't like the constraints imposed by the .NET Settings implementation, such as configuring user settings at compile time and using attributes.  After implementing my own key value store implementation I'm realizing I may use the .NET settings provider for application settings, since I want it strongly typed, to have default values, and to have validation, and my property bag like implementation for storing settings application extenders write. For example Jane user writes a plugin and can persist settings in the user directory of my application.  This scenario doesn't work using LocalFileSettingsProvider.

Here is my initial key value store implementation:

using System.Configuration;
using System.Xml;
 
public sealed class DebuggerSettings
{
    public string this[string propertyName]
    {
        get
        {
            var store = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.PerUserRoamingAndLocal);
            UserSettingsGroup values = (UserSettingsGroup)store.SectionGroups["userSettings"];
            if (values == null)
            {
                return null;
            }
            ClientSettingsSection myValues = (ClientSettingsSection)values.Sections[typeof(DebuggerSettings).FullName];
            if (myValues == null)
            {
                return null;
            }
            SettingElement setting = myValues.Settings.Get(propertyName);
            if (setting == null)
            {
                return null;
            }
            string returnValue = setting.Value.ValueXml.InnerText;
            return returnValue;
        }
        set
        {
            var store = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.PerUserRoamingAndLocal);
            UserSettingsGroup addSectionGroup = (UserSettingsGroup)store.SectionGroups["userSettings"];
            if (addSectionGroup == null)
            {
                addSectionGroup = new UserSettingsGroup();
                store.SectionGroups.Add("userSettings",addSectionGroup);
            }
            string sectionName = (typeof(DebuggerSettings).FullName);
            ClientSettingsSection clientSettingsSection = (ClientSettingsSection)addSectionGroup.Sections[sectionName];
            if (clientSettingsSection == null)
            {
                clientSettingsSection = new ClientSettingsSection();
                clientSettingsSection.SectionInformation.AllowExeDefinition = ConfigurationAllowExeDefinition.MachineToLocalUser;
                addSectionGroup.Sections.Add(sectionName, clientSettingsSection);
            }
            SettingElement addMe = new SettingElement(propertyName, SettingsSerializeAs.String);
            XmlElement element = new XmlDocument().CreateElement("value");
            element.InnerText = value;
            addMe.Value.ValueXml = element;
            clientSettingsSection.Settings.Add(addMe);
                
            store.Save();
        }
    }
}

Implementing System.Configuration.SettingsProvider

I wanted to store all settings for an application as user settings so my app wouldn't writing to the app.config and require administrative privileges.  I struggled figuring out how the components in System.Configuration worked together, but luckily Reflector, Visual Studio's .NET Framework Source debugging, ProcessMonitor, MSDN articles, StackOverflow posts, and Code Project articles helped get my scenario working.

During this process I learned the SettingsProvider only handles retrieving and saving values to the datastore and how to implement one.  I didn't end up using it since the LocalFileSettingsProvider met my needs, but the code below is a good starting point for someone wanting to implement settings persistence in a different manner:

using System;
using System.Collections.Specialized;
using System.Configuration;
using System.Diagnostics;
 
public sealed class CustomettingsProvider : SettingsProvider, IApplicationSettingsProvider
{
    NameValueCollection settingValues = new NameValueCollection();
 
    public override void Initialize(string name, NameValueCollection config)
    {
        Debug.WriteLine("in initialize override");
        base.Initialize(this.ApplicationName, settingValues);
    }
 
    /// <summary>
    /// MSDN states this property should be implemented with this getter and a do nothing setter.
    /// </summary>
    public override string ApplicationName
    {
        get  {  return (System.Reflection.Assembly.GetExecutingAssembly().GetName().Name); }
        set { Debug.WriteLine("set application name called"); }
    }
 
    public override SettingsPropertyValueCollection GetPropertyValues(SettingsContext context, SettingsPropertyCollection collection)
    {
        SettingsPropertyValueCollection returnValue = new SettingsPropertyValueCollection();
        foreach (SettingsProperty item in collection)
        {
            SettingsPropertyValue addMe = new SettingsPropertyValue(item);
            addMe.PropertyValue = String.Empty;
            returnValue.Add(addMe);
        }
 
        return returnValue;
    }
 
    public override void SetPropertyValues(SettingsContext context, SettingsPropertyValueCollection collection)
    {
        Debug.WriteLine("in setPropertyValues");
 
        foreach (SettingsPropertyValue item in collection)
        {
            bool isUserScoped = (item.Property.Attributes[typeof(UserScopedSettingAttribute)] is UserScopedSettingAttribute);
            bool isAppScoped = (item.Property.Attributes[typeof(ApplicationScopedSettingAttribute)] is ApplicationScopedSettingAttribute);
            if (isUserScoped && isAppScoped)
            {
                throw new ConfigurationErrorsException("Property can't be userScoped and appScoped according to msdn: http://msdn.microsoft.com/en-us/library/system.configuration.settingsprovider(VS.80).aspx");
            }
        }
    }
 
    public SettingsPropertyValue GetPreviousVersion(SettingsContext context, SettingsProperty property)
    {
        Debug.WriteLine("in getPreviousVersion");
        throw new NotSupportedException("Get Previous Version is not supported");
    }
 
    public void Reset(SettingsContext context)
    {
        Debug.WriteLine("in Reset");
    }
 
    public void Upgrade(SettingsContext context, SettingsPropertyCollection properties)
    {
        Debug.WriteLine("in Upgrade");
    }
}