OT : IIS State

Because the font is, occasionally, not full of crap...

I've got three web servers, all running IIS 4 on NT 4. I'm experiencing a
locking situation by the look of it where all servers go up to 60% average
CPU usage and stay there.

I've done a IISState on one of the servers and get the following (in order
of my interpretation) :

Thread ID: 47
System Thread ID: 185
Kernel Time: 0:0:5.875
User Time: 0:0:46.453
Thread Status: Thread is in a WAIT state.
Other information: Thread is waiting for a lock to be released. Looking for
lock owner.
Owning thread System ID: 1d4
Thread Type: ASP
Executing Page: Either ASP.dll is not being called on this thread or symbol
file not available. Unable to locate ASP page.
Continuing with other analysis.

# ChildEBP RetAddr
00 0364f7c4 77f6cfe2 ntdll!NtWaitForSingleObject+0xb
01 0364f834 77f67646 ntdll!RtlpWaitForCriticalSection+0xa5
02 0364f83c 77f64e24 ntdll!RtlEnterCriticalSection+0x46
03 0364f860 77b2102c ntdll!RtlFreeHeap+0x94
04 0364f870 653415ae ole32!CRetailMalloc_Free+0x17
05 0364f890 65341554 OLEAUT32!APP_DATA::FreeCachedMem+0x85
06 0364f8a0 653422e5 OLEAUT32!SysFreeString+0x57
07 0364f8b4 6b6010b3 OLEAUT32!VariantClear+0x72
WARNING: Stack unwind information not available. Following frames may be
wrong.
08 0364f8c8 6b601629 vbscript+0x10b3
09 01341c28 00000000 vbscript+0x1629


Several of these, all waiting on thread 1d4 - 1d4 is the following :

Thread ID: 29
System Thread ID: 1d4
Kernel Time: 0:0:4.359
User Time: 0:0:41.125
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
vbscript.dll -
Thread Type: ASP
Executing Page: Either ASP.dll is not being called on this thread or symbol
file not available. Unable to locate ASP page.
Continuing with other analysis.

# ChildEBP RetAddr
00 01a5f9fc 77f65296 ntdll!RtlpInsertFreeBlock+0xdf
01 01a5fa3c 77f64e6f ntdll!RtlpDeCommitFreeBlock+0x2d4
02 01a5fa68 77b2102c ntdll!RtlFreeHeap+0xdf
03 01a5fa78 653415ae ole32!CRetailMalloc_Free+0x17
04 01a5fa98 65341554 OLEAUT32!APP_DATA::FreeCachedMem+0x85
05 01a5faa8 653422e5 OLEAUT32!SysFreeString+0x57
06 01a5fabc 6b6010b3 OLEAUT32!VariantClear+0x72
WARNING: Stack unwind information not available. Following frames may be
wrong.
07 01a5fad0 6b601629 vbscript+0x10b3
08 013147b0 00000000 vbscript+0x1629

Now to me that looks like a simple ASP page running through a VBScript - why
would it be causing all the others to wait for it to complete?

I can post the full IISState log if needs be, but it generally repeats the
above.

 

*JEREMY*

 

You think it could be the regional setting then?

 

Heh - I've no idea - but I do propose we shout *JEREMY* whenever anyone
posts a computer-related technical query.

 

Only to ones I can answer.

 

<fx: scratches chin knowingly>

Hmmm, have you tried slapping the side of the box? Let me know if that
doesn't fix it - I have more.

 

Hmm - tough choice - slap it around, or spill coffee on it...

Drat, it's in a data centre so I can't do either

 

Damn. Gotta admit it, this one's got me beat. You're screwed. HTH,
etc..

 

<snip>

Is the script accessing references to apartment-threaded objects (such as
objects created from VB6 class modules) in Application or Session state?

I'm wondering if there's a deadlock when marshalling across apartment
boundaries.

 

Best bit of L1 analysis I've seen in ages. Cab, take a note.

The above is trying to free some allocation back and holding a lock
that the others are all waiting to get

It looks like it's running through a big long chain of free blocks to
deposit this newly freed allocation, given that you've caught it in:

If this list is huge or f**ked, then everyone is going to be sitting
around waiting for the lock. It's probably not circular or everything
would stop dear behind this.

I'll have a look tomorrow to see how the above works.

 

ewwww.

 

just the IIS upgrade will probably do the trick, I expect. They didn't
get it "right" until 5.

 

It's not actually the problem, but I appreciate your sentiments and agree
with you - however as you know you can't always get what you want in life.

Windows 2003 would be lovely - no one will pay for it.

Therefore, NT 4 because "it's practically free"

 

Are you taking the michael?

The MS groups are of the opinion it's to do with some excessive
concatenation going on, but I am slightly sceptical based on the behaviour.

 

LOL - if you're ever really bored I'll explain the whole arrangement to
you.. but the chances of someone "understanding"
are as about as likely as me getting my knee down.

 

No not really...I'd estimate <5% of 'the computer industry' can do
this stuff these days, mostly it's all politics and marketing now.

Actually, the answers below make more sense as it's probably fixed.
I'll have a look tomorrow to see what the above frames do.
I'd skip the w2k step, it's EOL anyway.

 

Here's some clues...

A variant is being cleared. Although it's a variant, it has a string value,
because SysFreeString is being called to clear it. SysFreeString is
attempting to return to the heap the memory that's been allocated for that
string value. RtlFreeHeap, the kernel mode heap deallocation routine,
employs a critical section to ensure that no two threads are attempting to
free heap-based memory at the same time. Thread 47 is hung up waiting to
enter this critical section.

Thread 29 appears to already be within the critical section, which is why
Thread 47 can't enter it. It's already in the kernel, releasing memory to
the heap.

Now, although you seem to have several threads waiting on thread 29, that's
just a snapshot at the point that you ran IISState. While it's possible that
thread 29 is actually holding everything up, it's much more likely that
there's LOTS of deallocation going on and the threads are tripping over each
other. Within a single process, only one thread can be executing within
RtlFreeHeap at a time. If there's lots of memory de-allocation going on,
they're going to be tripping over each other. Combine that with the fact
that there's going to be lots of corresponding memory allocation going on,
it really does sound like classic string concatenation. The fact that both
of these threads are in SysFreeString is further evidence to support it.

 

Only trouble is, if they recognise the support issue, go and buy three
upgrade licenses to Win2K3, take the servers down, rebuild them, bring them
back up, take them down again, fix them, bring them back up...they're still
going to have the same three servers locking up - 'cos it's almost certainly
application behaviour, not platform.

 

Steve Morgan wrote

Are you suggesting, without even knowing what they are, that the
application are causing a server to fail? Does that happen in Windows
Serverland then? No wonder people say it is shite.

 

Okay - and the solution would be..?
-Minimum of &'s
-Simplification of logic where applicable
Anything else?

What's sodding annoying is that :
a) I didn't write the web pages for the site but I now have to make it work
:-/
b) The old version, using 'crap' web classes worked perfectly well with far
more load.
c) We expressed doubts from the start to the 'customer' that the appliation
had ever been tested on a truly high level site.

On the plus side, I've increased my knowledge of IIS

Marketing People... FERITE!