After a recent SSRS upgrade from SQL 2008 to SQL 2008R2 Gold ( no patches) , we began to get complaints from users that reports were running slow and Internet Explorer was hanging up when certain reports were executed.

We ran some of our standard SSRS reports, and there were no unusual report failures. We looked at the SSRS logs and there was nothing which could explain the problem either.

We could reproduce the problem by running one of our SSRS reports – it seemed to run forever, when normally it would run in less than 10 seconds.

It seemed that every time we touched the report – things would hang up.  While the report was running, sometimes your laptop would go unresponsive as well. If you managed to click on any other tab in IE, the title bar for IE would include (Internet Explorer Unresponsive), and the entire  IE window would grey out.

Running the same report, with the same data, in Report Builder 3 was fine.

We ran Fiddler to see what we might learn by following the network traffic.  The times between send and receives were pretty fast. Summing up the times accounted for only a small portion of the 1:40 that it took to run the report.  However we did notice a pattern – there was a large wait on the client side. Fiddler looked like:

                Send -> receive

                Send -> receive

                Send -> receive

                <Long Wait>

                Send -> receive

 The sends come from the IE client, and the receive is the response from SSRS Reportserver.The client would send a request, get a quick response, then sometimes just wait.  Since the unexplained time was after a receive, and before a send, we know that the client is the problem. If the client would simply do the next send – all would be good.

 Running Task Manager on the client during the report run , we would see one or two processors using nearly 100% during the wait, then returning to normal. Below is task manager during the unresponsive period.

 

After a while we realized that the problem occurs when the parameters refresh. It took 1:40 to refresh the parameter list. There were several parameters, and simply choosing a different item from the parameter list would send us into this 1:40 wait.

 I pulled out the query which generates the parameter lists, and ran it in Management Studio – and it is fast, fast, fast.

 Running the report in the development environment also did not exhibit the same behavior, even though it was on the same version of SSRS as the problem prod environment.  There were lots of reports in prod that did not have this problem. We typically run 25,000 to 35,000 reports per day in production. Most were fine, but several reports, only in production, exhibited this behavior.

 Continuing to work, we discovered that the number of values in the parameter lists in the trouble prod reports were between 8K and 11k. The same report which did not exhibit this problem in dev, had  a much shorter list of values in its parameter lists. Changing the data source on the report in dev, to point to the production database – the dev report took forever to refresh, just like prod.  Now we know the problem is related to having a large number of values in the parameter list.  Checking other reports with the same problem, they all had a large parameter lists.

Microsoft has a KB article about this problem (http://support.microsoft.com/kb/2522708) and http://support.microsoft.com/kb/2506799/LN.  it is fixed in CU7 for SQL Server 2008 R2. This CU is not included in SP1, which is in beta right now. However, I assume it will be included in SP2 and later, when they come out.

 The KB article describes the problem to apply to “The report contains a large multi-select drop-down parameter list.”   Our reports with single select drop-downs also showed this problem, so it does not apply ONLY to multi-select.

Another thing to note is that the initial load of the parameters when the report first comes up in IE is fine. However when you change one of the parameter values, and the parameters refresh – you wait, wait, wait.

 

How do I know if I have this problem?

·         Report was created in an earlier version of SSRS than SQL 2008 R2

·         Initial report load, and execution time same as pre-upgrade.

·         Parameter refresh takes a long time, during which CPU utilization goes near 100%

·         You are running SQL Server 2008 R2 pre-CU7

 

We installed CU 7 and the 1:40 refresh time dropped to about 10 seconds, and processor utilization dropped – problem fixed. CU7 can be found at http://support.microsoft.com/kb/2489376.

 

Warning: I believe that installing SQL 2008 R2 CU7 breaks intellisense in Visual Studio 10. If so, there is a patch which fixes this also.

We recently upgraded SQL Server Reporting Services for a large SSRS user from SQL 2008 to SQL 2008R2 Gold  (no SQL 2008R2 patches).

Immediately some reports began to fail with a System.OverflowException.  Sometimes the report would run correctly. Other times the report would fail.  The implementation was 4 load balanced servers on the front end.  Failures were happening on all of the servers, so the problem wasn’t specific to a particular server.

The error that customers would see in SSRS is in the screen shot below:

 

Looking at the SSRS Error Logs, we found errors like these:

processing!WindowsService_0!165c!05/24/2011-06:00:18:: e ERROR: An exception has occurred in data set 'OutageDataSet'. Details: System.OverflowException: Value was either too large or too small for an Int32.

processing!WindowsService_0!165c!05/24/2011-06:00:18:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ProcessingAbortedException: , Microsoft.ReportingServices.ReportProcessing.ProcessingAbortedException: An error has occurred during report processing. ---> System.OverflowException: Value was either too large or too small for an Int32.

It was always System.OverflowException: Value was either too large or too small for an int32.

As it turns out, this is a known bug, documented in the KB article (http://support.microsoft.com/kb/2359606).

 There is a work around in the article which tells us to convert a grouping field from int to Cdbl or CLong. However, if you do this – you’ll have to figure out which item needs to be changed. The error message will give you the dataset, but not the column, nor the item within the report that exhibits the problem.

This is fixed in Cumulative Update 4(http://support.microsoft.com/kb/2345451) , which you can download and install immediately. This fix is also included in SQL 2008 R2 SP1, which at the time of this writing, is still in beta.

One of my friends and co-workers has been working on an SSRS in Sharepoint Integrated mode problem. Melissa Coates has an excellent blog post which details the issues.  Derek Sanderson also has a blog post about this issue.The short version is that images in reports in Sharepoint Integrated Mode can cause the report to run much more slowly than we though.  Think about a scorecard with the small images related to good, bad, growing, slowing etc. Each one of those images is a separate http Get - another round trip. In her example, Melissa has a dashboard type report which runs .5 seconds in native mode, and about 11 seconds in integrated mode.

Do some testing to ensure there are no surprises in your SSRS Sharepoint Integrated Mode installation.

To Melissa and others on her team who worked on this - great job!

We are going to follow up with more testing around this.... and we'll post the results.

This morning I got an email from Craig Ellis with a picture of us at PASS ( probably 2009 or 2008). It was titled "Me and the Prez". Craig is one of the many folks whom I have had the pleasure to know at PASS. It reminds me that PASS has been an anchor for me! PASS has been the source of many good friends. My career has been made better because of the things I learn at PASS, in fact - I truly believe that my life has been improved as a result. Exciting things are happening at PASS - stay tuned.

PASS is an organization which is composed of SQL people who simply wish for us all to be better, to do better, and to live better. While there have always been folks who would disparage PASS, I try not to let it bother me. Of course - I could be better. Of course - PASS could be better - duh!  However, there is no vast conspiracy, not even a small conspiracy, no evil empire, no secret society - it just ain't true - none of it! Never has been!

PASS is just you and me, trying to do as good as we can in a busy world!

I'm just a SQL guy - a country boy from Charlotte, NC. Most of the folks I have come in contact with are just the same - folks simply trying to do their job, trying to do good for themselves, their company and their family. They volunteer for PASS because it is good for all of us - helping someone else is the best way to help yourself. To all of you who have volunteered for PASS, led a chapter, helped with a committee - ( even the program committee which did not accept my session this year) - I say - Thanks!. Thanks for all of us. And Thanks from me. 

The PASS conference is often the best week of the year for me!.  I get to learn stuff, and visit with all of my friends. It is always a great time! See you the first week of November!

"Me and the Prez" - just makes me feel a little bit sentimental!

I recently read a research report from the ITIC which which studied the number of security flaws reported against different databases since 2002. The ITIC is the Information Technology Intelligence Consulting. It is located in Boston and its bio says it is an independent research and consulting firm that covers high technology.

The research states that, since 2002, Microsoft SQL Server has had the fewest reported number of vulnerabilities of any major database platform - only 49. These statistics were reported from the NIST ( National Institute of Standards and Technology) which is a government monitoring agency.

Oracle has reported a whopping 321 flaws, more than 6 time that of SQL Server. So when someone talks about how great Oracle is compared with SQL Server, or complains about security patches from Microsoft - this research may come in handy. I guess this the result of the Trustworthy Computing initiative which begain in 2002.  You may recall the SQL Slammer worm which really messed us all up in about May of 2002. Afterwards, the SQL Server group stopped all new development and spent months going through existing code with the purpose of making it safer and more secure.  Perhaps this is the payoff - good job SQL TEAM!

 

Security Vulnerabilities since 2002

SQL Server	49
MySQL	98
IBM DB2	121
Oracle	321

 

The entire report can be seen  at http://itic-corp.com/blog/2010/09/sql-server-most-secure-database-oracle-least-secure-database-since-2002/

BIDS

Takeaway:

Details:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" ToolsVersion="2.0">
  <State>$base64$PFNvdXJjZUNvbnRyb2xJbmZvIHhtbG5zOnhzZD0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTg==</State>
  <DataSources />
  <DataSets />
  <Reports>
    <ProjectItem>
      <Name>Report1.rdl</Name>
      <FullPath>Report1.rdl</FullPath>
    </ProjectItem>
  </Reports>
  <Configurations>
    <Configuration>
      <Name>Debug</Name>
      <Platform>Win32</Platform>
      <Options>
        <OutputPath>bin\Debug</OutputPath>
        <TargetServerVersion>SSRS2008R2</TargetServerVersion>
        <TargetFolder>TiffTest</TargetFolder>
        <TargetDataSourceFolder>Data Sources</TargetDataSourceFolder>
        <TargetDatasetFolder>Datasets</TargetDatasetFolder>
        <TargetReportPartFolder>Report Parts</TargetReportPartFolder>
      </Options>
    </Configuration>
    <Configuration>
      <Name>DebugLocal</Name>
      <Platform>Win32</Platform>
      <Options>

Technet says (http://technet.microsoft.com/en-us/library/ee635898.aspx) and below:

Important
If you save a SQL Server 2008 Report Server project in the SQL Server 2008 R2 version Business Intelligence Development Studio you can no longer open it in the SQL Server 2008 version of Business Intelligence Development Studio.

Reports

Takeaway:

Reports can be deployed to SQL 2008 or R2, but unsupported features will be stripped away.

Details:

Error level	Description
0	Most severe and unavoidable build issues that prevent preview and deployment of reports.
1	Severe build issues that change the report layout drastically.
2	Less severe build issues that change report layout insignificantly.
3	Minor build issues that change the report layout in minor ways that might not be noticeable.
4	Used only for publishing warnings

 

You can set the ErrorLevel in a project configuration, as below:

It feels strange that my first blog post will not be technical. Technical will come soon.  I work for a great company Mariner, based in Charlotte, NC. Mariner is an ALL BI - ALL THE TIME company where you can be challenged with good work and encouraged and expected to keep up to date with all the latest MS Stack based Business Inteliigence practices and software.

Mariner is looking for excellent skills SSIS, SSAS, SSRS, SQL to join the team that I work in. We are getting new projects and wish to add some folks. The team you will work with are motivated, smart, hard-working people, who enjoy their job and their lives.  If you are really good, we'd love you to interview to join us.

Mariner is a small company (30-ish), but well respected and specialized.Our home is in Charlotte, NC.  We do what we say - both for our customers and co-workers. If you want to be the best, then come work with the best. We are looking for people right now!!  http://www.mariner-usa.com/jobs/