In the last SQL Server users group meeting (#117) on April 2^nd I presented a session about FileTables – a new exciting feature in SQL Server 2012 that enables applications to manage structured as well as unstructured data seamlessly.

Here are the materials from my presentation:

On February 29^th we organized a SQL Server event – The Best of SQL Server – together with Media-Tech College. This event was the opening shot for our new partnership with Media-Tech College. We plan to provide together high quality professional training solutions in the field of Microsoft SQL Server to businesses and individuals in the north. The event was very successful with around 50 attendees and with 3 excellent presentations.

Yariv Inbar (Media-Tech) opened the event and talked about our new exciting partnership and our plans for the future, such as: professional courses, community events, hands-on seminars, and more. I presented a session about advanced query processing, in which I spoke about various aspects of parameterization in SQL Server. Assaf Shalem (TopTix) introduced Extended Events and presented some practical troubleshooting scenarios. Shy Engelberg (Madeira) presented an overview of the new exciting features in SQL Server 2012.

Here are all the materials from the presentations:

I hope you enjoyed the event. I'll be happy to hear your feedback…

Thanks,
Guy

On February 20^th I attended the Microsoft event – A Journey into SQL Server 2012. The event was very interesting, and included many great sessions…

Roy Pasternak (Microsoft) presented a short overview of the new features in SQL Server 2012. There are so many of them. SQL Server 2012 is truly a big advancement in all areas of the product – mission-critical confidence, dynamic development, breakthrough insight and cloud-ready. More information can be found here.

Yossi Elkayam (Microsoft) then drilled down into the new BI features, like Power View and Data Quality Services. There are really so many…

Danny Ravid (Brillix) presented the new high availability solution in SQL Server 2012 – AlwaysOn. This is one of the most exciting new features in SQL Server 2012, in my opinion. Microsoft took the best parts of Failover Clustering and Database Mirroring, added some more, and came up with a robust platform that enables true high availability as well as load balancing at the database level. With this platform we can have multiple readable replicas of a group of databases with a single virtual name for the application. More information can be found here.

Itai Binyamin (Veracity) presented the new tool for developers in SQL Server 2012 – SQL Server Data Tools. The SSDT brings together all the great functionality from Visual Studio 2010 and SQL Server Management Studio, and it creates a unified and rich development environment for application developers and database developers alike. And it's part of the SQL Server product stack, so you get it for free.

Ami Levin (DBSophic) presented some new T-SQL features, like metadata discovery, paging and the new Sequence object.

Rachel Yehezkel (Microsoft) presented a short overview of the SQL Azure platform – Microsoft's data platform in the cloud. If you have questions about SQL Azure, then there is a new forum in Hebrew about Windows Azure and SQL Azure. You can find it here.

I presented a session about FileTables – a new exciting feature in SQL Server 2012 that enables applications to manage structured as well as unstructured data seamlessly. I plan to write more about FileTables in the near future. In the meanwhile, here are the materials from my presentation:

I hope you enjoyed the event in general and my presentation in particular. I'll be glad to hear your feedback…

Thanks,
Guy

Sometimes we want to populate a table with random data, mainly for testing purposes. For example, let's consider the following table:

CREATE TABLE
    Sales.Orders
(
    Id            INT             NOT NULL    IDENTITY(1,1) ,
    CustomerId    INT             NOT NULL ,
    DateAndTime   DATETIME2(0)    NOT NULL ,
    OrderStatus   NVARCHAR(50)    NULL
);
GO

Let's say we want to insert 100,000 rows with random data. We want to generate random values between 1 and 1,000,000 for the "CustomerId" column, and we want to generate random date & time values in the past 5 years for the "DateAndTime" column. For now, let's ignore the "OrderStatus" column. Here is how I would do it:

INSERT INTO
    Sales.Orders WITH (TABLOCK)
(
    CustomerId ,
    DateAndTime
)
SELECT TOP (100000)
    CustomerId    = ABS (CHECKSUM (NEWID ())) % 1000000 + 1 ,
    DateAndTime   = DATEADD (SECOND , - ABS (CHECKSUM (NEWID ())) % (60 * 60 * 24 * 365 * 5) , SYSDATETIME ())
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2;
GO

Notice a few things:

1. I'm using the TABLOCK hint in order to use minimal logging. For more information about minimally logged inserts, I recommend this article by Itzik Ben-Gan.
2. I'm retrieving rows from sys.all_columns cross joined to itself. It really doesn’t matter what table/s I'm using in this case. All I need is to make sure that the data set contains at least 100,000 rows. The TOP clause limits the number of rows returned to 100,000.
3. I'm using the NEWID function in order to generate random values. If you try to use the RAND function, it will return the same value for the whole recordset, because the RAND function is evaluated only once for the query. If you want a different value for each row (of course you do), then you need to use the NEWID function.

Here is how the table looks like (10 first rows) after the insert:

Now, let's add the "OrderStatus" column to the game. This column should contain one of the following values: "New", "Open", "Closed", "Cancelled" and "Archived". Generating random values for this column is a bit trickier. We can achieve this by using the CASE statement. But first, a few words about this statement…

The CASE statement has two flavors. One is called "Simple CASE", and it compares a single expression to a set of simple expressions. The second is called "Searched CASE", and it evaluates a set of general Boolean expressions.

Here is an example of a simple CASE:

CASE CustomerTypeId
    WHEN 1    THEN N'Regular'
    WHEN 2    THEN N'VIP'
    WHEN 3    THEN N'New'
    WHEN 4    THEN N'Blocked'
    ELSE N'Unknown'
END

And here is an example of a searched CASE:

CASE
    WHEN Price <= $100.00                       THEN N'Cheap'
    WHEN Price > $100.00 AND ProductType = 1    THEN N'Medium'
    WHEN Price > $100.00 AND ProductType = 2    THEN N'Expensive'
    WHEN Price IS NULL                          THEN N'Unknown Price'
    ELSE N'Unknown Product Type'
END

In both cases, the first condition that evaluates to TRUE determines the result. The ELSE clause is optional. If all conditions evaluate to FALSE and there is no ELSE clause, then the result is NULL.

Now, let's go back to our example. In order to generate a random value for the "OrderStatus" column, we can generate a random value between 1 and 5, and use the simple CASE statement in order to translate these numbers into text values, like this:

TRUNCATE TABLE
    Sales.Orders;
GO

INSERT INTO
    Sales.Orders WITH (TABLOCK)
(
    CustomerId ,
    DateAndTime ,
    OrderStatus
)
SELECT TOP (100000)
    CustomerId    = ABS (CHECKSUM (NEWID ())) % 1000000 + 1 ,
    DateAndTime   = DATEADD (SECOND , - ABS (CHECKSUM (NEWID ())) % (60 * 60 * 24 * 365 * 5) , SYSDATETIME ()) ,
    OrderStatus   = CASE ABS (CHECKSUM (NEWID ())) % 5 + 1
                        WHEN 1    THEN N'New'
                        WHEN 2    THEN N'Open'
                        WHEN 3    THEN N'Closed'
                        WHEN 4    THEN N'Cancelled'
                        WHEN 5    THEN N'Archived'
                    END
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2;
GO

Here are the results (first 10 rows):

Notice that there are many NULL values in the "OrderStatus" column. This is strange, cause the expression we used (ABS (CHECKSUM (NEWID ())) % 5 + 1) always returns a number between 1 and 5, so one of the conditions in the CASE clause should have been evaluated to TRUE. What’s going on?

Well, I'm glad you asked, because this is what I wanted to blog about this time. SQL Server doesn't really have two flavors of the CASE statement. It's actually two interfaces for our convenience. When we use the simple CASE statement, SQL Server translates it into the searched CASE format, because this is actually the only format that exists for the query processor. So the previous INSERT statement is translated by SQL Server to this:

TRUNCATE TABLE
    Sales.Orders;
GO

INSERT INTO
    Sales.Orders WITH (TABLOCK)
(
    CustomerId ,
    DateAndTime ,
    OrderStatus
)
SELECT TOP (100000)
    CustomerId    = ABS (CHECKSUM (NEWID ())) % 1000000 + 1 ,
    DateAndTime   = DATEADD (SECOND , - ABS (CHECKSUM (NEWID ())) % (60 * 60 * 24 * 365 * 5) , SYSDATETIME ()) ,
    OrderStatus   = CASE
                        WHEN ABS (CHECKSUM (NEWID ())) % 5 + 1 = 1    THEN N'New'
                        WHEN ABS (CHECKSUM (NEWID ())) % 5 + 1 = 2    THEN N'Open'
                        WHEN ABS (CHECKSUM (NEWID ())) % 5 + 1 = 3    THEN N'Closed'
                        WHEN ABS (CHECKSUM (NEWID ())) % 5 + 1 = 4    THEN N'Cancelled'
                        WHEN ABS (CHECKSUM (NEWID ())) % 5 + 1 = 5    THEN N'Archived'
                    END
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2;
GO

Now you can see that the expression ABS (CHECKSUM (NEWID ())) % 5 + 1 is calculated separately for each condition in the Searched CASE statement. For the first condition, a random value between 1 and 5 is calculated and compared to "1". The probability of a match is 20%. If it's FALSE, then we move one to the second condition, for which another (different) random value between 1 and 5 is calculated and, this time, compared to "2". Again, we have a probability of 20% for a match, and so on. It is very likely that all conditions will evaluate to FALSE, and in that case the result will be NULL. Actually, the probability of getting NULL from this CASE expression is 0.85 = 32.768%. If you count the number of NULL values in the table, you'll find that there are around 32,768 such rows (out of 100,000 rows in total).

OK, so we learned that the Simple CASE is actually a Searched CASE behind the scenes, and that for this reason, using a random expression with the Simple CASE is problematic. That's good to know (I think), but I still want to populate the "OrderStatus" column with random values. How do I do that?

Well, I'm glad you asked again. There are several options to solve this problem. One of them is to use a small table with a single column containing the 5 text values, cross join to this table as well and sort the whole recordset randomly. Like this:

CREATE TABLE
    Sales.OrderStatuses
(
    OrderStatus NVARCHAR(50) NOT NULL
);
GO

INSERT INTO
    Sales.OrderStatuses
(
    OrderStatus
)
VALUES
    (N'New') ,
    (N'Open') ,
    (N'Closed') ,
    (N'Cancelled') ,
    (N'Archived');
GO

TRUNCATE TABLE
    Sales.Orders;
GO

INSERT INTO
    Sales.Orders WITH (TABLOCK)
(
    CustomerId ,
    DateAndTime ,
    OrderStatus
)
SELECT TOP (100000)
    CustomerId    = ABS (CHECKSUM (NEWID ())) % 1000000 + 1 ,
    DateAndTime   = DATEADD (SECOND , - ABS (CHECKSUM (NEWID ())) % (60 * 60 * 24 * 365 * 5) , SYSDATETIME ()) ,
    OrderStatus   = OrderStatus
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2
CROSS JOIN
    Sales.OrderStatuses
ORDER BY
    NEWID () ASC;
GO

This method works, but it's very inefficient, because the query processor has to complete all the join operations first, then calculate the NEWID function for every row, then sort the whole recordset according to the values of NEWID, and only then retrieve the top 100,000 rows. Since the recordset is huge (around 134,000,000 in my case), the sort operation is going to take forever. You can try to replace sys.all_columns with another table that has fewer rows in order to get a recordset of a size close to 100,000, but it's still not going to be an easy task.

Another solution to the problem is to insert the 100,000 rows one by one in a loop. Inside the loop we can calculate the random value once and put it in a variable, and then we can use the Simple CASE statement with the variable. Like this:

DECLARE
    @Counter                    AS INT        = 100000 ,
    @RandomValueBetween1And5    AS TINYINT;

TRUNCATE TABLE
    Sales.Orders;

WHILE
    @Counter > 0
BEGIN

    SET @RandomValueBetween1And5 = ABS (CHECKSUM (NEWID ())) % 5 + 1;

    INSERT INTO
        Sales.Orders WITH (TABLOCK)
    (
        CustomerId ,
        DateAndTime ,
        OrderStatus
    )
    SELECT TOP (100000)
        CustomerId    = ABS (CHECKSUM (NEWID ())) % 1000000 + 1 ,
        DateAndTime   = DATEADD (SECOND , - ABS (CHECKSUM (NEWID ())) % (60 * 60 * 24 * 365 * 5) , SYSDATETIME ()) ,
        OrderStatus   = CASE @RandomValueBetween1And5
                            WHEN 1    THEN N'New'
                            WHEN 2    THEN N'Open'
                            WHEN 3    THEN N'Closed'
                            WHEN 4    THEN N'Cancelled'
                            WHEN 5    THEN N'Archived'
                        END;

    SET @Counter -= 1;

END;
GO

This method also works, of course, but it is also inefficient, because there are 100,000 INSERT operations instead of one.

There are more inefficient ways to solve this problem, but let's look at an efficient one. Let's go back to the original query with the Simple CASE statement. We can still use the same query, but we need to make sure that the expression in the Simple CASE statement is calculated only once, and then compared to the other values (1 through 5). We can achieve this by performing the calculation of the random value in a derived table, like this:

TRUNCATE TABLE
    Sales.Orders;
GO

INSERT INTO
    Sales.Orders WITH (TABLOCK)
(
    CustomerId ,
    DateAndTime ,
    OrderStatus
)
SELECT TOP (100000)
    CustomerId    = ABS (CHECKSUM (NEWID ())) % 1000000 + 1 ,
    DateAndTime   = DATEADD (SECOND , - ABS (CHECKSUM (NEWID ())) % (60 * 60 * 24 * 365 * 5) , SYSDATETIME ()) ,
    OrderStatus   = CASE RandomValueTable.RandomValue
                        WHEN 1    THEN N'New'
                        WHEN 2    THEN N'Open'
                        WHEN 3    THEN N'Closed'
                        WHEN 4    THEN N'Cancelled'
                        WHEN 5    THEN N'Archived'
                    END
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2
CROSS JOIN
    (
        SELECT
            RandomValue = ABS (CHECKSUM (NEWID ())) % 5 + 1
    )
    AS
        RandomValueTable;
GO

Now the random value is calculated once inside the derived table, and it is exposed to the outer query as a column. When SQL Server transforms the Simple CASE statement into a Searched CASE format, it's still OK, because the "RandomValue" column is not calculated separately for each condition. It's actually similar to the solution using a loop and a variable to hold the random value. Here, we use a column in a derived table to hold the random value instead of a variable.

Now we have a solution that works and is also very efficient. Here are the results:

A few weeks ago Kalen Delaney presented a 3-day seminar in Israel. Unfortunately, I wasn't able to attend the seminar, but some of our consultants attended and returned from the seminar with some interesting insights.

The first day was about concurrency control. Among other things, Kalen spoke about the snapshot isolation level and how row versioning works when this isolation level is used.

One question was remained unanswered. Suppose you have an index on column "Col1", and you update this column while a snapshot transaction is open. The index key is, of course, updated and moved to the new position in the index, and it also now has an additional 14-bytes pointer to the versioning info in tempdb. As long as queries in the snapshot transaction access the table, they will retrieve the old value from the version store. Let's assume you updated the column from 2 to 5. Now, what happens if a query in the snapshot transaction tries to retrieves the rows with Col1 = 2? The information about the previous value (2) is stored in tempdb, and the pointer is stored in the index under the new value (5). So if a query performs an index seek in order to retrieve only the rows with a value of 2, how would SQL Server know about the old value stored in tempdb?

Let's look at an example. First, here is the setup:

USE

master;

GO

– Create the SnapshotTest database

IF

DB_ID (N'SnapshotTest') IS NOT NULL

BEGIN

ALTER DATABASE

SnapshotTest

SET

SINGLE_USER

WITH

ROLLBACK IMMEDIATE;

DROP DATABASE

SnapshotTest;

END;

GO

CREATE DATABASE

SnapshotTest;

GO

ALTER DATABASE

SnapshotTest

SET

ALLOW_SNAPSHOT_ISOLATION ON;

GO

– Create the "dbo.Test" table

USE

SnapshotTest;

GO

CREATE TABLE

dbo.Test

(

Col1 INT NOT NULL ,

Col2 CHAR(5) NOT NULL

);

GO

– Populate the "dbo.Test" table with 10 consecutive rows

INSERT INTO

dbo.Test WITH (TABLOCK)

(

Col1 ,

Col2

)

SELECT TOP (10)

Col1 = ROW_NUMBER () OVER (ORDER BY (SELECT NULL) ASC) ,

Col2 = N'XXXXX'

FROM

sys.all_columns;

GO

SELECT

Col1 ,

Col2

FROM

dbo.Test

ORDER BY

Col1 ASC;

GO

– Create a non-clustered index on the "Col1" column

CREATE NONCLUSTERED INDEX

ix_Test_nc_nu_Col1

ON

dbo.Test (Col1 ASC);

GO

Let's look at the index page before we update the table. First, we use DBCC IND in order to retrieve the index pages:

DBCC IND (N'SnapshotTest' , N'dbo.Test' , 2);

GO

Here are the results (I left only the interesting columns):

As we can see, there is a single index page (89) that contains the index data. Page #90 is the IAM page, so it's not relevant for our test. Let's look at the contents of page #89:

DBCC TRACEON (3604);

GO

DBCC PAGE (N'SnapshotTest' , 1 , 89 , 1);

GO

Here are the results (only a partial view):

Slot 0, Offset 0×60, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×000000000FAFA060

0000000000000000: 06010000 004f0000 00010000 00††††††††…..O…….

Slot 1, Offset 0×6d, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×000000000FAFA06D

0000000000000000: 06020000 004f0000 00010001 00††††††††…..O…….

Slot 2, Offset 0×7a, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×000000000FAFA07A

0000000000000000: 06030000 004f0000 00010002 00††††††††…..O…….

Slot 3, Offset 0×87, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×000000000FAFA087

0000000000000000: 06040000 004f0000 00010003 00††††††††…..O…….

Slot 4, Offset 0×94, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×000000000FAFA094

0000000000000000: 06050000 004f0000 00010004 00††††††††…..O…….

Slot 5, Offset 0xa1, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×000000000FAFA0A1

0000000000000000: 06060000 004f0000 00010005 00††††††††…..O…….

We can see the first 6 slots in the index page containing the values 1 through 6. I marked the values in red. Each slot is an "INDEX_RECORD", and the size of each record is 13 bytes. So far so good…

Now, let's begin our experiment. In a new connection, we are going to start a snapshot transaction and retrieve the data from the "dbo.Test" table:

USE

SnapshotTest;

GO

SET TRANSACTION ISOLATION LEVEL SNAPSHOT;

GO

BEGIN TRANSACTION

ReadData;

GO

SELECT

Col1 ,

Col2

FROM

dbo.Test

ORDER BY

Col1 ASC;

GO

And now, in the first connection, let's update Col1 from 2 to 5:

UPDATE

dbo.Test

SET

Col1 = 5

WHERE

Col1 = 2;

GO

If we retrieve the data again inside the snapshot transaction (in the second connection), we will still see the previous value (2), as expected:

SELECT

Col1 ,

Col2

FROM

dbo.Test

ORDER BY

Col1 ASC;

GO

If we try to retrieve only the rows with Col1 = 2, it also works:

SELECT

Col1 ,

Col2

FROM

dbo.Test

WHERE

Col1 = 2;

GO

How does it work?

Let's look again at the contents of the index page:

Slot 0, Offset 0×60, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×00000000111EA060

0000000000000000: 06010000 004f0000 00010000 00††††††††…..O…….

Slot 1, Offset 0xe2, Length 27, DumpStyle BYTE

Record Type = GHOST_INDEX_RECORD Record Attributes = VERSIONING_INFO Record Size = 27

Memory Dump @0×00000000111EA0E2

0000000000000000: 4a020000 004f0000 00010001 00602d00 †J….O…….`-.

0000000000000010: 00010001 009e0100 000000†††††††††††††………..

Slot 2, Offset 0×7a, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×00000000111EA07A

0000000000000000: 06030000 004f0000 00010002 00††††††††…..O…….

Slot 3, Offset 0×87, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×00000000111EA087

0000000000000000: 06040000 004f0000 00010003 00††††††††…..O…….

Slot 4, Offset 0xfd, Length 27, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = VERSIONING_INFO Record Size = 27

Memory Dump @0×00000000111EA0FD

0000000000000000: 46050000 004f0000 00010001 00000000 †F….O……….

0000000000000010: 00000000 009e0100 000000†††††††††††††………..

Slot 5, Offset 0×94, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×00000000111EA094

0000000000000000: 06050000 004f0000 00010004 00††††††††…..O…….

Slot 6, Offset 0xa1, Length 13, DumpStyle BYTE

Record Type = INDEX_RECORD Record Attributes = Record Size = 13

Memory Dump @0×00000000111EA0A1

0000000000000000: 06060000 004f0000 00010005 00††††††††…..O…….

First, we see that the value "2" has been changed to "5" and moved to its new position. We can also see that it now has the "VERSIONING_INFO" attribute, and it contains the additional 14 bytes for the pointer to the version store (27 bytes instead of 13). This is how SQL Server accesses the old value (2) when the snapshot transaction retrieves all the rows.

But we can also see that there is a ghost index record where the old record was, with the value "2". The ghost record always appears when an index record is deleted, but it is usually cleaned up by the ghost cleanup task. In our case, the ghost record won't be removed, because it also has the "VERSIONING_INFO" attribute, and there is an open snapshot transaction that requires that data. It points to the same entry in the version store. This record is used in order to retrieve the old value (2) when we try to retrieve only the rows with Col1 = 2. As soon as we commit or rollback the snapshot transaction, the ghost record will be cleaned up.

Let's use an example. We'll create the "Inventory.Products" table and populate it with million rows of random data:

Notice that this table has a clustered index on the "Id" column. Currently this is the only index we have. No non-clustered indexes at all.

Now, let's look at a query that uses the AND operator:

If we run this query, we'll get a clustered index scan, as expected:

If you're using SQL Server 2008 or later, then you'll probably also see the optimizer's suggestion for a missing index. The optimizer suggests creating a non-clustered index on the "Status", "Weight" and "ListPrice" columns, and also including the "Id" and "Name" columns. There are some problems with the optimizer's suggestion and with the missing indexes feature in general, but this is a subject for another post. For now, let's listen to the optimizer's suggestion and create the index:

Now the execution plan looks like this:

Here is the execution plan:

Here is the execution plan:

Now the plan looks like that:

Let's start all over again, this time with a query that uses the OR operator:

We already have the index on the "ListPrice" column, but if we run this query, we get the following execution plan:

The execution plan is still the same:

And now look at the execution plan:

Look how the optimizer performs 3 different index seek operations, one for each index, and then combines the results using the "Concatenation" operator. It then performs an aggregate operation in order to remove duplicate rows. The number of logical reads is 1,213, which is around 12% of the 10,085 logical reads when scanning the whole table. This is very good, taking into account that the query returns 191,527 rows, which is around 19% of the million rows in the whole table.

Here is the full script that I used in this post:

3 months ago I presented a session in the special Rosh HaShana user group meeting. The session was about Monitoring Page Splits. I talked about the difference between a "real" page split (mid-page split) and a page allocation (end-page split). I demonstrated that all the tools available by SQL Server to monitor page splits actually monitor both types (mid-page and end-page), and there is no way to monitor only "real" page splits, which are a common concern for DBAs.

After the presentation, Ami Levin posted a question about it in the MVP forum and connected between me and Jonathan Kehayias. Jonathan suggested a much better solution, which is still based on the "LOP_DELETE_SPLIT" operation in the transaction log. But instead of searching for it in the log using "sys.fn_dblog", he suggested to monitor it with Extended Events using the "sqlserver.transaction_log" event, which is a new event in SQL Server 2012.

There are several advantages to this approach over the "sys.fn_dblog" approach:

1. The "sqlserver.transaction_log" event is fired as the log record is created, and the relevant events are put in memory. This is much more efficient than reading the transaction log from the log file.

2. With Extended Events, you can control the timeframe in which you would like to monitor page splits by starting and stopping the event session. With "sys.fn_dblog", the amount of log records you'll read is determined by the last time the log was truncated (for example, by a CHECKPOINT operation or by a log backup operation).

3. The Extended Events infrastructure lets you correlate between transaction log records and the operations that caused these records to be created. This is a very powerful tool, and I plan to write about it more in the near future.

4. While the content of the transaction log still remains undocumented, Extended Events is a well-documented and supported mechanism. The "sys.fn_dblog" function isn't.

Jonathan has just written a great post describing his solution. You can read it here.

I would like to thank Jonathan for sharing his solution with me (and with everyone, actually), and I would like to thank Ami for making the connection.

In my previous post, Concatenating Strings, I promised to post my own version of the PRINT statement. It's a stored procedure called "PrintMax". The reason I use this stored procedure is because the PRINT statement can't handle a string with more than 4,000 bytes.

Here is a quote from Books Online:

"A message string can be up to 8,000 characters long if it is a non-Unicode string, and 4,000 characters long if it is a Unicode string. Longer strings are truncated. The varchar(max) and nvarchar(max) data types are truncated to data types that are no larger than varchar(8000) and nvarchar(4000)."

Sometimes, you want to print a long string. For example, you might want to print the definition of a long stored procedure. Or you might have a very long dynamic batch that you are going to execute, but you want to print it first for debug purposes.

The problem with the PRINT statement is not only that it prints up to the first 4,000 bytes. It also truncates your text without even generating a warning.

So I use my version of the PRINT statement – the "PrintMax" stored procedure – which does the following:

1. It accepts a NVARCHAR(MAX) input parameter. So any text value you pass (Unicode or non-Unicode, fixed length or variable length) is converted to NVARCHAR(MAX).

2. It splits the large text into chunks of up to 4,000 characters.

3. It searches for the last line break within each chunk and splits the string there. This way, it doesn’t just cut sentences in the middle. It uses existing line breaks in the text in order to perform the split. The output is much more readable.

You can download the "PrintMax" stored procedure from here.

Now, let's try it…

First, let's try to print a long string of X's with both methods – the regular PRINT statement and the "PrintMax" stored procedure:

DECLARE @nvcLargeText AS NVARCHAR(MAX);

SET @nvcLargeText = REPLICATE (CAST (N'X' AS NVARCHAR(MAX)) , 5000);

SELECT LargeTextLength = LEN (@nvcLargeText);

PRINT @nvcLargeText;

EXECUTE dbo.PrintMax @inLargeText = @nvcLargeText;

As you can see, the PRINT statement truncates the text at 4,000 characters. The "PrintMax" stored procedures splits the text into two parts. It prints the first 4,000 characters in the first line, and the additional 1,000 characters in the second line.

Now, let's try to print the largest SQL module in your database:

DECLARE @nvcLargeText AS NVARCHAR(MAX);

SELECT TOP (1) @nvcLargeText = definition

FROM sys.sql_modules

ORDER BY LEN (definition) DESC;

SELECT LargeTextLength = LEN (@nvcLargeText);

PRINT @nvcLargeText;

EXECUTE dbo.PrintMax @inLargeText = @nvcLargeText;

See how the PRINT statement truncates the code in the middle without even warning you about it? And did you see how the "PrintMax" stored procedure prints the whole code nicely?

This script creates the "PrintMax" stored procedure, which prints a text of any size. It's intended to replace the PRINT statement, which is limited to a text size of up to 4,000 bytes.

The "PrintMax" stored procedure does the following:

1. It accepts a NVARCHAR(MAX) input parameter. So any text value you pass (Unicode or non-Unicode, fixed length or variable length) is converted to NVARCHAR(MAX).

2. It splits the large text into chunks of up to 4,000 characters.

3. It searches for the last line break within each chunk and splits the string there. This way, it doesn’t just cut sentences in the middle. It uses existing line breaks in the text in order to perform the split. The output is much more readable.

For more information about this stored procedure: http://www.madeira.co.il/all-you-can-print/.

Yesterday I needed to run a long batch inside a string using sys.sp_executesql. So I declared a variable with the NVARCHAR(MAX) data type, assigned the text of the batch into it, and tried to execute it. When I assigned the text into the variable, I used concatenation of a few strings. Something like that:

DECLARE @Statement AS NVARCHAR(MAX);

SET @Statement = N'Long String…' + @Variable + N'Another Long String…' + …

I have probably done that a million times before. But this time I received an error. The error resulted from the fact that my string was truncated. The length of my string should have been more than 10,000 bytes. I checked the length of the @Statement variable using the DATALENGTH function, and it turned out to be only 8,000 bytes.

I guess it's the first time that I'm trying to assign such a long string to a NVARCHAR(MAX) variable. At first, I couldn't figure it out. What's the whole point of the NVARCHAR(MAX) data type, if such a variable cannot contain more than 8,000 bytes? It doesn't make sense…

I thought that maybe there was something about my text that caused this strange behavior, so I decided to try something much simpler:

DECLARE @String AS NVARCHAR(MAX);

SET @String = REPLICATE (N'X' , 5000);

SELECT DATALENGTH (@String);

I expected the length to be 10,000 bytes, but it was 8,000 bytes again. At this point I started to get nervous. It says "MAX", doesn't it?

I checked the value of @@TEXTSIZE, but it was set to the maximum value (2147483647). I played with it for a while, and after some (wasted) time, I finally understood what's going on. So here it is…

Every expression in SQL Server has a data type. This includes a column in a table, a variable, a returned value from a scalar function, and even a literal, such as the expression N'ABCD'. In the case of the first 3 types of expressions, the data type is well defined. But in the case of a literal, it's a little less clear what the data type should be. SQL Server has its own rules as of what data type to assign to a literal. So what's the data type of N'ABCD'?

I can think of several options that would make sense, but according to SQL Server's rules, the data type of N'ABCD' is NVARCHAR(4). As long as the number of characters (n) in the string literal is less or equal to 4,000, the data type would be NVARCHAR(n). If the number of characters is greater than 4,000, the data type would be NVARCHAR(MAX). So far it makes sense, doesn't it?

Now, let's talk about concatenation. What do you think would be the data type of the following expression?

N'ABCD' + N'EFG'

Correct! It's NVARCHAR(7). Everything is making sense. Isn't it great?

Now, let's complicate things just a little bit. Suppose you have an expression that is a concatenation of two string literals – one of them contains 3,000 characters and the other contains 2,000 characters. Something like that:

What would be the data type in this case? Well, it can't be NVARCHAR(5000), so it should be NVARCHAR(MAX), right? Wrong! The data type of this expression is NVARCHAR(4000), and the last 1,000 characters simply get truncated. So if you assign this expression to a NVARCHAR(4000) variable, the value of the variable would be 3,000 X's and 1,000 Y's. No errors, no warnings.

If you assign it to a NVARCHAR(MAX) variable, then the following things will happen. First, SQL Server will perform the concatenation and assign the data type NVARCHAR(4000) to the expression (and truncate the last 1,000 characters). Then, SQL Server will implicitly convert the NVARCHAR(4000) expression to NVARCHAR(MAX) and assign it to the variable. So the type of the variable would be NVARCHAR(MAX), but its value would still be 3,000 X's and 1,000 Y's.

This explains the behavior that I originally experienced. Since I concatenated a few long strings, the result was truncated to accommodate 4,000 characters and then assigned to my NVARCHAR(MAX) variable. That's great, isn't it?

So how do we solve this problem? If we concatenate a NVARCHAR(MAX) expression with any other string expression, the result would be NVARCHAR(MAX). It doesn't matter how many expressions we want to concatenate. As long as at least one of them would be NVARCHAR(MAX), the result would be NVARCHAR(MAX) as well. So all we need to do is to explicitly convert one of the literals to NVARCHAR(MAX), like this:

The data type of this expression is NVARCHAR(MAX), and its length is 5,000 characters, as expected. Not so pretty, but it works.

Remember the REPLICATE (N'X' , 5000) example? Why is the data type of this expression NVARCHAR(4000) and not NVARCHAR(MAX)? Well, the answer is simple – The REPLICATE function simply concatenates the same expression the requested number of times. It's just like the following expression:

N'X' + N'X' + N'X' + …

And we already know what happens here…

By the way, if you try to print a NVARCHAR(MAX) expression using the PRINT statement, it will also truncate the value to 4,000 characters, even if the expression is a true NVARCHAR(MAX) with more than 4,000 characters. This has nothing to do with the explanation given above. It happens because the PRINT statement can't handle more than 4,000 Unicode characters or 8,000 non-Unicode characters. So if it's given a NVARCHAR(MAX) value, it truncates it to NVARCHAR(4000). Again, no errors, no warnings.

This is why I use my own version of PRINT, which is a stored procedure called "PrintMax". I'll talk about it next time…