Tuesday, July 6, 2010

Debug = true


Its been ages since I wrote something useful. It was time to break the shackles and dust off my blog. So after a long hiatus I start over again. This one pertains to common problems observed in ASP.Net Application bottlenecks and issues.

Microsoft Developer Support or ("CSS" - Customer Support Services) is where you're sent within Microsoft when you've got problems. They see the most interesting bugs, thousands of issues and edge cases and collect piles of data. They report this data back to the ASP.NET team (and other teams) for product planning. With all those cases and all the projects, there's basically two top things that cause trouble in production ASP.NET web sites. Long story short, Debug Mode and Anti-Virus software.

The excerpts for this post has been taken from MSDN content and other insightful blog posts.


#1 Issue - Configuration

Seems the #1 issue in support for problems with ASP.NET 2.x and 3.x is configuration.

Symptoms

Notes

  • OOM
  • Performance
  • High memory
  • Hangs
  • Deadlocks

There are more debug=true cases than there should be.

People continue to deploy debug versions of their sites to production. We can automatically transform the web.config and change it to a release version. More on that later.

Additionally, if you leave debug=true on individual pages, note that this will override the application level setting.

Here's why debug="true" is bad:

  • Overrides request execution timeout making it effectively infinite
  • Disables both page and JIT compiler optimizations
  • In 1.1, leads to excessive memory usage by the CLR for debug information tracking
  • In 1.1, turns off batch compilation of dynamic pages, leading to 1 assembly per page.
  • For VB.NET code, leads to excessive usage of WeakReferences (used for edit and continue support).

An important note: Contrary to what is sometimes believed, setting retail="true" in a element is not a direct antidote to having debug="true"!

#2 Issue - Problems with an External (non-ASP.NET) Root Cause

Sometimes when you're having trouble with an ASP.NET site, the problem turns out to not be ASP.NET itself. Here's the top three issues and their causes. This category are for cases that were concluded because of external reasons and are outside of the control of support to directly affect. The sub categories are 3rd party software, Anti-virus software, Hardware, Virus attacks, DOS attacks, etc.

If you've ever run a production website you know there's always that argument about whether to run anti-virus software in production. It's not like anyone's emailing viruses and saving them to production web servers, but you want to be careful. Sometimes IT or security insists on it. However, this means you'll have software that is not your website software trying to access files at the same time your site is trying to access them.

Here's the essence as a bulleted list

  • Concurrency while under pressure: This causes problems in big software. Make sure your anti-virus software is configure appropriately and that you're aware of which processes are accessing which files, as well as how, why and when
  • Profile your applications: .NET and the Web are not black boxes. You can see what's happening if you look. Know what bytes are going out the wire. Know who is accessing the disk. Measure twice, cut once, they say? I say measure a dozen times. You'd be surprised how often folks put an app in production and they've never once profiled it.
  • Anti-Virus Software: It can't be emphasized enough that site owners should ensure they are running the latest AV engine and definitions from their chosen anti-malware vendor. They've seen people hitting hangs due to flakey AV drivers that are over two years out of date. Another point about AV software is that it is not just about old-school AV scanning of file access. Many products now do low level monitoring of port activity, script activity within processes and memory allocation activity and do not always do these things 100% correctly. Stay up to date!
  • Know where you're calling out to: Also, connection to remote endpoints: calling web services, accessing file systems etc. All of this can slow you down if you're not paying attention. Is your DNS correct? Did you add your external hosts to a hosts file to remove DNS latency?
  • processModel autoconfig=true: This is in machine.config and people mess with it. Don't assume that you know better than the defaults. Everyone wants to change the defaults, add threads, remove threads, change the way the pool works because they think their textboxes-over-data application is special. Chances are it's not, and you'd be surprised how often people will spend days on the phone with support and discover that the defaults were fine and they had changed them long ago and forgotten. Know what you've changed away from the defaults, and know why.

...and here's the table of details:

Issue

Product

Description

Symptoms

Notes

Anti-virus software

All

Anti-virus software is installed onto Servers and causes all kinds of problems.

  • Application restarting
  • Slow performance
  • Session variable are null
  • Cannot install hotfix
  • Intermittent time outs
  • High memory
  • Session lost
  • IDE Hangs
  • Deadlocks

This consists of all AV software reported by our customers. All cases do not report the AV software that is being used so the manufacturer is not always known.

KB821438, KB248013,KB295375, KB817442

3rd party Vendors

All

This is a category of cases where the failure was due to a 3rd party manufacturer.

  • Crash
  • 100% CPU
  • High memory
  • Framework errors
  • Hang

The top culprits are 3rd party database systems, and 3rd party internet access management systems.

Microsoft component

All

Microsoft software

  • Intermittent time outs
  • High memory
  • Deadlocks
  • 100% CPU
  • Crash

Design issues that cause performance issues like sprocs, deadlocks, etc. Profile your applications and the database! (Pro tip: select * from authors doesn't scale.) Pair up DBAs and programmers and profile from end to end.

Hopefully this post at least gets us started

No comments: