Building A WMI Performance Data Warehouse To Identify IIS Capacity Bottlenecks

0

No comments posted yet

Comments

brianpaulflynn (3 years ago)

Some great linkes that address when you should and should not use asynchronous programming. http://weblogs.asp.net/rajbk/archive/2010/04/18/asp-net-asynchronous-pages-and-when-to-use-them.aspx http://blogs.msdn.com/b/tmarq/archive/2010/04/14/performing-asynchronous-work-or-tasks-in-asp-net-applications.aspx http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx

brianpaulflynn (3 years ago)

I've also posted the code on CodePlex : http://wmidw.codeplex.com/

brianpaulflynn (3 years ago)

I've posted code samples on github. The sample is a console application. It can be converted into a Windows service with a little effort. It has been done but the console app is a simple demonstration. https://github.com/brianpaulflynn/WMI-Performance-Counter-Logging-Console-App

Slide 15

ASP.NET v2.050727\Request Wait Time – The number of milliseconds the most recent request was waiting in the queue.

Slide 16

ASP.NET v2.050727\Request Execution Time –The number of milliseconds that it took to execute the most recent request.

Slide 17

How many requests can you be executing before you hit a ceiling? What happens when down stream dependencies stall the completion of requests? What can be done about it?

Slide 19

Two types of queuing Global Queuing should stay near zero under load Application Queuing Should at zero unless thread starvation occurs

Slide 20

Requires programming which may take a lot of time. Can add a lot of capacity! Making calls asynchronously adds some programming complexity and resource overhead that costs a little time to execute. If the downstream calls always run quickly, the additional overhead may not be justified. The overall performance may have been better implemented synchronously.

Slide 21

Quick to implement but adds limited capacity. If a bunch of threads get blocked then suddenly become unblocked, there can be a rush on system resources to process them all in parallel. This could result in a lot of CPU context switching and deep CPU queues. It may have been better to allow application queuing to prevent so many requests from processing simultaneously.

Slide 22

The number of native OS threads created and owned by the CLR to act as underlying threads for .NET thread objects. This counters value does not include the threads used by the CLR in its internal operations; it is a subset of the threads in the OS process. Following instances of this counter can reveal how close each worker process is to thread capacity.

Slide 26

The number of threads in the processor queue. There is a single queue for processor time even on computers with multiple processors. If the CPU is very busy (90 percent and higher utilization) and the PQL average is consistently higher than 2 per processor, you may have a processor bottleneck that could benefit from additional CPUs.

Slide 27

This counter displays the amount of virtual memory (in bytes) currently committed by the Garbage Collector. (Committed memory is the physical memory for which space has been reserved on the disk paging file). Following this counter can reveal how close a worker process is to it’s memory cap as defined in the IIS application.

Slide 28

If Disk IO and memory utilization are high, there may be paging occurring. Adding memory may help unless the program is too hungry or leaking memory.

Slide 1

Building A WMI Performance Data Warehouse To Identify IIS Capacity Bottlenecks By Brian Flynn Information Technology Renaissance Man http://www.linkedin.com/in/brianpaulflynn

Slide 2

The Mysterious Internals of IIS

Slide 3

Windows Management Instrumentation aka “WMI”, a set of extensions to the Windows Driver Model that provides an operating system interface through which instrumented components provide information and notification. http://en.wikipedia.org/wiki/Windows_Management_Instrumentation

Slide 4

Performance Monitor & WMI

Slide 5

Accessing WMI w/ VB Script

Slide 6

Accessing WMI w/ VB.NET

Slide 7

Star Schema Data Warehouse

Slide 8

Obtaining A Data Set

Slide 9

Pivot The Data with Excel

Slide 10

Current Connections Web Service\Current Connections

Slide 11

Request Status

Slide 12

Current Requests ASP.NET v2.05727\Requests Current

Slide 13

Executing Requests ASP.NET Apps v2.05727\Requests Executing

Slide 14

Queued Requests ASP.NET v2.05727\Requests Queued

Slide 15

Wait Times ASP.NET v2.050727\Request Wait Time

Slide 16

Execution Times ASP.NET v2.050727\Request Execution Time

Slide 17

Planning For & Monitoring Capacity

Slide 18

Worker Process Restaurant Analogy 1. Take Order 2. Submit Order 3. Pick Up Order 4. Deliver Order 5. Take Order 6. Submit Order 7. Pick Up Order 8. Deliver Order 9. Take Order 10. Submit Order 11. Pick Up Order 12. Deliver Order WAIT WAIT WAIT

Slide 19

A Glass Ceiling : Worker Process Thread Starvation An ASP.NET worker process will not execute more than ((maxWorkerThreads*number of CPUs)-minFreeThreads).

Slide 20

Programming Solution : Asynchronous Request Processing 1. Take Order 2. Submit Order 7. Pick Up Order 8. Deliver Order 3. Take Order 4. Submit Order 9. Pick Up Order 10. Deliver Order 5. Take Order 6. Submit Order 11. Pick Up Order 12. Deliver Order

Slide 21

Configuration Solution : Worker Process Thread Pool Tweaking

Slide 22

Monitoring WP Thread Pools .NET CLR LocksAndThreads\# of current physical Threads

Slide 23

The Application Queues Should Stay At Zero! ASP.NET Apps v2.05727\Requets In Application Queue The application queue, or virtual directory queue, which is managed by the HttpRuntime class. The HttpRuntime class provides run-time services for ASP.NET applications. There is one queue for each virtual directory. The application queue is configured in Machine.config by the <httpRuntime appRequestQueueLimit> property. If it is ever greater than zero, thread starvation has occurred.

Slide 24

Scaling Beyond The Worker Process : Farms, Gardens & CPUs… Farm Thread Capacity = { [ (maxWorkerThreads) * (# CPUs) ] – (minFreeThreads) } * (# Worker Processes) * (# Hosts) Note : NLB “No Affinity” is important for evenly distributing requests across farm nodes.

Slide 25

So Now What? What if requests and threads are piling up? System resources might be a bottleneck : CPU Memory Disk Network We can track correlations to those too!

Slide 26

Processor Queue Length System\Processor Queue Length

Slide 27

Memory Utilization .NET CLR Memory\# Total committed Bytes

Slide 28

Disk Queue Length Physical Disk\Current Disk Queue Length

Slide 29

Network Queue Length Network Interface\Output Queue Length

Slide 30

Recap WMI is a treasure trove of diagnostic data. IIS may not provide interfaces to obtain details regarding request processing but with a little effort you can create your own. Asynchronous programming can significantly enhance the capacity of a web application but the programming can take a lot of time. The capacity of a single ASP.NET worker process can be quickly enhanced by tweaking the thread pool via the machine.config.

Slide 31

References Using Performance Monitor WMI WMI Code Creator Improving .NET Application Performance and Scalability Chapter 6 – Improving ASP.NET Performance Chapter 15 – Measuring .NET Application Performance Chapter 17 — Tuning .NET Application Performance ASP.NET Performance Monitoring, and When to Alert Administrators How To: Monitor the ASP.NET Thread Pool Using Custom Counters Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications ASP.NET Request Queues Hangs and how to solve them - part 2 - Queuing

Summary: How to dissect IIS request processing activity and identify performance bottlenecks. This technique can be used to positively diagnose thread starvation resulting from downstream resource delays and what can be done to resolve that by adding capacity, resilience, high availability and scalability. Code samples can be found on github. https://github.com/brianpaulflynn/WMI-Performance-Counter-Logging-Console-App

URL:
More by this User
Most Viewed