ASP.NET v2.050727\Request Wait Time – The number of milliseconds the most recent request was waiting in the queue.

ASP.NET v2.050727\Request Execution Time –The number of milliseconds that it took to execute the most recent request.

How many requests can you be executing before you hit a ceiling? What happens when down stream dependencies stall the completion of requests? What can be done about it?

Two types of queuing Global Queuing should stay near zero under load Application Queuing Should at zero unless thread starvation occurs

Requires programming which may take a lot of time. Can add a lot of capacity! Making calls asynchronously adds some programming complexity and resource overhead that costs a little time to execute. If the downstream calls always run quickly, the additional overhead may not be justified. The overall performance may have been better implemented synchronously.

Quick to implement but adds limited capacity. If a bunch of threads get blocked then suddenly become unblocked, there can be a rush on system resources to process them all in parallel. This could result in a lot of CPU context switching and deep CPU queues. It may have been better to allow application queuing to prevent so many requests from processing simultaneously.

The number of native OS threads created and owned by the CLR to act as underlying threads for .NET thread objects. This counters value does not include the threads used by the CLR in its internal operations; it is a subset of the threads in the OS process. Following instances of this counter can reveal how close each worker process is to thread capacity.

The number of threads in the processor queue. There is a single queue for processor time even on computers with multiple processors. If the CPU is very busy (90 percent and higher utilization) and the PQL average is consistently higher than 2 per processor, you may have a processor bottleneck that could benefit from additional CPUs.

This counter displays the amount of virtual memory (in bytes) currently committed by the Garbage Collector. (Committed memory is the physical memory for which space has been reserved on the disk paging file). Following this counter can reveal how close a worker process is to it’s memory cap as defined in the IIS application.

If Disk IO and memory utilization are high, there may be paging occurring. Adding memory may help unless the program is too hungry or leaking memory.

Building A WMI Performance Data Warehouse To Identify IIS Capacity Bottlenecks By Brian Flynn Information Technology Renaissance Man http://www.linkedin.com/in/brianpaulflynn

The Mysterious Internals of IIS

Windows Management Instrumentation aka “WMI”, a set of extensions to the Windows Driver Model that provides an operating system interface through which instrumented components provide information and notification. http://en.wikipedia.org/wiki/Windows_Management_Instrumentation

Performance Monitor & WMI

Accessing WMI w/ VB Script

Accessing WMI w/ VB.NET

Star Schema Data Warehouse

Obtaining A Data Set

Pivot The Data with Excel

Current Connections Web Service\Current Connections

Request Status

Current Requests ASP.NET v2.05727\Requests Current

Executing Requests ASP.NET Apps v2.05727\Requests Executing

Queued Requests ASP.NET v2.05727\Requests Queued

Wait Times ASP.NET v2.050727\Request Wait Time

Execution Times ASP.NET v2.050727\Request Execution Time

Planning For & Monitoring Capacity

Worker Process Restaurant Analogy 1. Take Order 2. Submit Order 3. Pick Up Order 4. Deliver Order 5. Take Order 6. Submit Order 7. Pick Up Order 8. Deliver Order 9. Take Order 10. Submit Order 11. Pick Up Order 12. Deliver Order WAIT WAIT WAIT

A Glass Ceiling : Worker Process Thread Starvation An ASP.NET worker process will not execute more than ((maxWorkerThreads*number of CPUs)-minFreeThreads).

Programming Solution : Asynchronous Request Processing 1. Take Order 2. Submit Order 7. Pick Up Order 8. Deliver Order 3. Take Order 4. Submit Order 9. Pick Up Order 10. Deliver Order 5. Take Order 6. Submit Order 11. Pick Up Order 12. Deliver Order

Configuration Solution : Worker Process Thread Pool Tweaking

Monitoring WP Thread Pools .NET CLR LocksAndThreads\# of current physical Threads

The Application Queues Should Stay At Zero! ASP.NET Apps v2.05727\Requets In Application Queue The application queue, or virtual directory queue, which is managed by the HttpRuntime class. The HttpRuntime class provides run-time services for ASP.NET applications. There is one queue for each virtual directory. The application queue is configured in Machine.config by the <httpRuntime appRequestQueueLimit> property. If it is ever greater than zero, thread starvation has occurred.

Scaling Beyond The Worker Process : Farms, Gardens & CPUs… Farm Thread Capacity = { [ (maxWorkerThreads) * (# CPUs) ] – (minFreeThreads) } * (# Worker Processes) * (# Hosts) Note : NLB “No Affinity” is important for evenly distributing requests across farm nodes.

So Now What? What if requests and threads are piling up? System resources might be a bottleneck : CPU Memory Disk Network We can track correlations to those too!

Processor Queue Length System\Processor Queue Length

Memory Utilization .NET CLR Memory\# Total committed Bytes

Disk Queue Length Physical Disk\Current Disk Queue Length

Network Queue Length Network Interface\Output Queue Length

Recap WMI is a treasure trove of diagnostic data. IIS may not provide interfaces to obtain details regarding request processing but with a little effort you can create your own. Asynchronous programming can significantly enhance the capacity of a web application but the programming can take a lot of time. The capacity of a single ASP.NET worker process can be quickly enhanced by tweaking the thread pool via the machine.config.

References Using Performance Monitor WMI WMI Code Creator Improving .NET Application Performance and Scalability Chapter 6 – Improving ASP.NET Performance Chapter 15 – Measuring .NET Application Performance Chapter 17 — Tuning .NET Application Performance ASP.NET Performance Monitoring, and When to Alert Administrators How To: Monitor the ASP.NET Thread Pool Using Custom Counters Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications ASP.NET Request Queues Hangs and how to solve them - part 2 - Queuing