In my previous post, I've demonstrated how Foreign keys can help the optimizer make better decisions when creating an execution plan. However, I've also mentioned that FKs make inserts and updates run slower. In this short post I'll show the actual impact the FK has on these actions.

Why does the insert run slower ?

A foreign key is basically a constraint. It limits the data in the referencing column to a list of values found in the referenced table. This makes SQL Server check the data before inserting it, to make sure the constraint is not violated. This check takes time, but how much time ?

To demonstrate I've created 2 tables: NameList (populated with 100 Ids and names) and DataTable that will reference the NameList table.

At first I've created the DataTable without the FK.

-- Create the list table
CREATE TABLE
    dbo.NameList
(
    Id        INT            NOT NULL ,
    Name    VARCHAR(10)    NOT NULL ,

    CONSTRAINT pk_NameId_c PRIMARY KEY CLUSTERED (Id)
);

INSERT INTO
    dbo.NameList(Id, Name)
SELECT TOP 100
    ROW_NUMBER()OVER(ORDER BY T1.name) ,
    'Name_'+CAST(ROW_NUMBER()OVER(ORDER BY T1.name) AS VARCHAR(4))
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2;
GO

-- Create data table
CREATE TABLE
    dbo.DataTable
(
    NameId INT NOT NULL
);
GO

I've also created a third table (RandomData) from which data will be selected into the DataTable. This table contains 1,000,000 records that match the values in the list table.

-- Create random data
SELECT TOP 1000000
    ABS(CHECKSUM(NEWID())%100)+1 AS RandId
INTO
    dbo.RandomData
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2;
GO

Inserting data without a FK:

First lets see how long it takes to insert 1,000,000 records into a table without a FK. To test this, I've run an insert statement 10 times and measured the AVG run time.

-- Insert with out a FK and check time
IF EXISTS (SELECT * FROM sys.foreign_keys
            WHERE object_id = OBJECT_ID(N'dbo.fk_DataList')
            AND parent_object_id = OBJECT_ID(N'dbo.DataTable'))
BEGIN
    ALTER TABLE dbo.DataTable
    DROP CONSTRAINT fk_DataList;
END;
GO

DECLARE
    @StartTime    DATETIME2 ,
    @EndTime    DATETIME2 ,
    @i            INT = 1,
    @n            INT = 10;

TRUNCATE TABLE dbo.DataTable;

SET @StartTime  = SYSDATETIME();

WHILE @i <= @n
BEGIN
    INSERT INTO dbo.DataTable (NameId)
    SELECT RandId FROM dbo.RandomData;

    SET @i+=1;
END;

SET @EndTime = SYSDATETIME();

SELECT Avg_Insert_Time = CAST(CAST(DATEDIFF(MS,@StartTime,@EndTime) AS DECIMAL)/1000/@n AS DECIMAL(7,3));
GO

I run this test 12 time, and eliminated the best and worst results. The average run time was 4.68 sec.

The execution plan for the insert was very simple.

Inserting with a FK:

I run the same test, only this time I've created a FK before.

-- Add the FK
TRUNCATE TABLE dbo.DataTable;

ALTER TABLE dbo.DataTable
ADD CONSTRAINT fk_DataList FOREIGN KEY (NameId)
REFERENCES dbo.NameList (Id);
GO

-- Now insert with a FK and check time
DECLARE
    @StartTime    DATETIME2 ,
    @EndTime    DATETIME2 ,
    @i            INT = 1,
    @n            INT = 10;

TRUNCATE TABLE dbo.DataTable;

SET @StartTime  = SYSDATETIME();

WHILE @i <= @n
BEGIN
    INSERT INTO dbo.DataTable (NameId)
    SELECT RandId FROM dbo.RandomData;

    SET @i+=1;
END;

SET @EndTime = SYSDATETIME();

SELECT Avg_Insert_Time = CAST(CAST(DATEDIFF(MS,@StartTime,@EndTime) AS DECIMAL)/1000/@n AS DECIMAL(7,3));
GO

This time the average run time was 6.30sec. Almost 35% more. I must say I didn't think the impact is that significant.

The execution plan this time was very different. The data is sorted, and the list table is accessed as well.

Enjoying both worlds:

As we just saw, a FK can have significant impact on performance during inserts. Sometimes it's a price we are willing to pay, but sometimes our system is overloaded as it is. What can we do to minimize the impact on inserts while still keeping the FK ?

One solution is to drop the FK right before the insert and then rebuild it. This way the insert itself is performed as fast as possible while we still keep our data consistent.

-- Dropping the FK before the insert and recrating it after it's finished
DECLARE
    @StartTime    DATETIME2 ,
    @EndTime    DATETIME2 ,
    @i            INT = 1,
    @n            INT = 10;

TRUNCATE TABLE dbo.DataTable;

SET @StartTime  = SYSDATETIME();

WHILE @i <= @n
BEGIN
    ALTER TABLE dbo.DataTable
    DROP CONSTRAINT fk_DataList;

    INSERT INTO dbo.DataTable (NameId)
    SELECT RandId FROM dbo.RandomData;

    ALTER TABLE dbo.DataTable
    ADD CONSTRAINT fk_DataList FOREIGN KEY (NameId)
    REFERENCES dbo.NameList (Id);

    SET @i+=1;
END;

SET @EndTime = SYSDATETIME();

SELECT Avg_Insert_Time = CAST(CAST(DATEDIFF(MS,@StartTime,@EndTime) AS DECIMAL)/1000/@n AS DECIMAL(7,3));
GO

This time, the average run time was 5.03 sec, only 2.4% more than without using a FK. The only "danger" is this way there is a possibility that data that violates the FK can be inserted and we won't be able to rebuild the FK. In order to avoid this we could simply use a try-catch and rollback in case to FK cannot be rebuilt.

]]> http://www.madeira.co.il/foreign-keys-impact-on-inserts/feed/ 0 http://www.madeira.co.il/foreign-keys-and-how-they-help-the-optimizer/ http://www.madeira.co.il/foreign-keys-and-how-they-help-the-optimizer/#comments Fri, 17 Feb 2012 14:26:31 +0000 ‫Shay Attiya‬ http://www.madeira.co.il/?p=4295

I love foreign keys. In my opinion they just make perfect sense.

They keep my data “clean” and consistent. Whenever two tables need to correspond with each other, I like to make sure that they do. For me as a DBA, the easiest and most intuitive way to do that is to create a FK constraint. Some of you (maybe even most of you) will say that FKs can be replaced by the application. Let it do all of the “dirty work” of making sure that the data in our DB makes sense business wise. But if there’s something I hate, it’s cleaning up after some programmer’s bug. Data fixes are a pain in the ***.

However, there is another reason why FKs are important. They can help the optimizer make better and more efficient decisions when creating an execution plan. To explain exactly how and why, let first see what a FK is.

What’s a Foreign Key ?

A foreign key represents a relation between two tables in a database. The relation is determined by matching the value in a column (or combination of columns) in the referencing table to the value\s in the column\s of the referenced table. The value\s in the referenced table must be unique, either a primary key or a unique constraint.

These 2 conditions make sure 2 things will always be true:

1) For every value in the FK there is a matching record in the referenced table

2) There is only one match

How does it help the optimizer ?

Naturally, the optimizer knows how to take these 2 trues into consideration when creating the execution plan. To demonstrate I’ve created 2 tables with a FK relation between them.

The NameList table contains 100 names and Ids with the Id as a primary key for the table.

-- Create the list tables
CREATE TABLE
    dbo.NameList
(
    Id    INT            NOT NULL ,
    Name  VARCHAR(10)    NOT NULL ,

    CONSTRAINT pk_NameId_c PRIMARY KEY CLUSTERED (Id)
);

INSERT INTO
    dbo.NameList(Id, Name)
SELECT TOP 100
    ROW_NUMBER()OVER(ORDER BY T1.name) ,
    'Name_'+CAST(ROW_NUMBER()OVER(ORDER BY T1.name) AS VARCHAR(4))
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2;
GO

The RandomNames table contains 100,000 records with 4 columns. The Id is the PK for the table. The other 3 columns hold values between 1 and 100 and the values are exactly the same for all 3. The difference between them is that the first column is constrained by a FK, the second column isn’t, and the third also constrained by a FK but is nullable.

-- Create the data table
CREATE TABLE
    dbo.RandomNames
(
    Id            INT    NOT NULL IDENTITY(1,1) ,
    NameId        INT    NOT NULL ,
    NameId_NoFK   INT    NOT NULL ,
    NameId_NULL   INT    NULL ,

    CONSTRAINT pk_RandNameId_c PRIMARY KEY CLUSTERED (Id) ,

    CONSTRAINT fk_NameID FOREIGN KEY (NameId)
    REFERENCES dbo.NameList (Id) ,

    CONSTRAINT fk_NameID_NULL FOREIGN KEY (NameId_NULL)
    REFERENCES dbo.NameList (Id)
);

INSERT INTO
    dbo.RandomNames
(
    NameId ,
    NameId_NoFK ,
    NameId_NULL
)
SELECT
    RandId ,
    RandId ,
    RandId
FROM(
    SELECT TOP 100000
        ABS(CHECKSUM(NEWID())%100)+1 AS RandId
    FROM
        sys.all_columns AS T1
    CROSS JOIN
        sys.all_columns AS T2) AS T;
GO

Now, let’s join the 2 tables, every time using a different column (NameId, NameId_NoFK, NameId_NULL), and see how the execution plan differs for every query. Notice that I’m only selecting values from the RandomNames table.

SELECT TOP 100
    RN.Id
FROM
    dbo.RandomNames AS RN
INNER JOIN
    dbo.NameList AS NL
    ON RN.NameId = NL.Id;
GO

SELECT TOP 100
    RN.Id
FROM
    dbo.RandomNames AS RN
INNER JOIN
    dbo.NameList AS NL
    ON RN.NameId_NoFK = NL.Id;
GO

SELECT TOP 100
    RN.Id
FROM
    dbo.RandomNames AS RN
INNER JOIN
    dbo.NameList AS NL
    ON RN.NameId_NULL = NL.Id;
GO

When using the FK column the optimizer recognizes that the inner join in meaningless in this query. Because there must be a match and only one match, every record in the RandomNames must return and no duplicate can be created, so the optimizer simple scans the table. When using the NoFK column the optimizer has no way of knowing this and must perform the join. Notice that the nullable column behaves just like the NoFK column. This is because the optimizer can’t say every row as a match so the column is treated like it doesn’t have a FK (at least when it comes to executing the query).

Now you might say a better query would eliminate the need for the join. If you know your schema well enough you can just SELECT TOP 100 Id FROM dbo.RandomNames; and get the same result (that’s what the optimizer did), but a lot of times the person writing the query doesn’t know the schema very well. Other time the query is already written for us in a form of a view.

CREATE VIEW
    dbo.VW_RandomNames
AS
    SELECT
        RN.Id ,
        NL.Name
    FROM
        dbo.RandomNames AS RN
    INNER JOIN
        dbo.NameList AS NL
        ON RN.NameId = NL.Id;
GO

CREATE VIEW
    dbo.VW_RandomNames_NoFK
AS
    SELECT
        RN.Id ,
        NL.Name
    FROM
        dbo.RandomNames AS RN
    INNER JOIN
        dbo.NameList AS NL
        ON RN.NameId_NoFK = NL.Id;
GO

CREATE VIEW
    dbo.VW_RandomNames_NULL
AS
    SELECT
        RN.Id ,
        NL.Name
    FROM
        dbo.RandomNames AS RN
    INNER JOIN
        dbo.NameList AS NL
        ON RN.NameId_NULL = NL.Id;
GO

Now when we use the view, there is no way of rewriting the query.

SELECT TOP 100 Id
FROM dbo.VW_RandomNames;
GO

SELECT TOP 100 Id
FROM dbo.VW_RandomNames_NoFK;
GO

SELECT TOP 100 Id
FROM dbo.VW_RandomNames_NULL;
GO

Good thing the optimizer’s there to help us out. The execution plans are just like before.

The FK is also helpful when we simply want to filter out records that don’t have a match.

SELECT TOP 100 Id
FROM dbo.RandomNames
WHERE NameId IN (SELECT Id FROM dbo.NameList);
GO

SELECT TOP 100 Id
FROM dbo.RandomNames
WHERE NameId_NoFK IN (SELECT Id FROM dbo.NameList);
GO

SELECT TOP 100 Id
FROM dbo.RandomNames
WHERE NameId_NULL IN (SELECT Id FROM dbo.NameList);
GO

Again, the optimizer knows there is no need to check the NameList table when a FK is present (and it’s not nullable).

However, an odd thing happened when I tried to run these queries with a little change. I used NOT IN instead of IN to get only the records that don’t have a match.

SELECT TOP 100 Id
FROM dbo.RandomNames
WHERE NameId NOT IN (SELECT Id FROM dbo.NameList);
GO

SELECT TOP 100 Id
FROM dbo.RandomNames
WHERE NameId_NoFK NOT IN (SELECT Id FROM dbo.NameList);
GO

SELECT TOP 100 Id
FROM dbo.RandomNames
WHERE NameId_NULL NOT IN (SELECT Id FROM dbo.NameList);
GO

I thought that the optimizer would immediately recognized that when a FK is present this is not a possible scenario and would automatically return an empty record set without scanning any table. I was wrong…

The optimizer still goes over both tables and there is no difference between the FK column and the NoFK column. It gets even worse in the nullable FK column. It’s clear that the optimizer made a wrong decision, and I must say I couldn’t figure out why. My only guess is that the optimizer is not very effective when NULL values might be involved.

Bottom line

FK can help the optimizer. They make queries run faster and prevent unnecessary reads (less shared locks). This is true as long as no NULL values are involved. You can avoid that by using an Id that is not used instead of using NULL (-1 for example). That will eliminate the need for a nullable FK column.

It’s true that FKs make inserts and updates run slower, so the overall benefits of FKs are not that great performance wise. But we are still left with my main reason for using FKs. Consistent, “Clean” data.

As I’ve said before… “Data fixes are a pain in the ***”.

Today I want to talk about Log Shipping.

Log Shipping is an easy way to implement a high availability solution. It uses SQL Server’s transaction log to synchronize a stand by server. The configuration of the log shipping is really easy to set up as I’m about to show.

Basic requirements:

1) The Primary Server must be in “Full” or “Bulk-Logged” recovery model

2) The Secondary Server must be from the same version as the primary (or later) and can be either in “Standby” or “No Recovery” mode. When using the Standby mode, the server can be used for reporting purposes, but it won’t be accessible during the log shipping process.

3) A third monitor server can be used but this is optional.

Step 1 – Configuring the Primary Server:

Right click on the DB you wish to log ship and go to “Tasts -> Ship Transaction Logs…”. Click on “Enable this as a primary database in a log shipping configuration” check box and press on the “Backup Settings…” button.

The “Transaction Log Backup Settings” window will open. In it you will have to specify the path to the folder where the transaction log backups will be saved. Here you can also determine the frequency of transaction log backups (the default is every 15 minutes).

Step 2 – Configuring the Secondary Server:

After the primary server is configured, it’s time to configure the secondary server by pressing the “Add” button. More than one secondary server can be defined to create multiple standby servers.

First, you’ll have to connect to the secondary server.

In the “Initialize Secondary Database” tab you can choose the way you wish to initialize the secondary database (this is pretty self-explanatory).

In the “Copy Files” tab you’ll need to specify the folder where you wish to save the log backups shipped from the primary server, and the frequency of copying between the servers.

In the “Restore Transaction Log” tab you can select the mode you want the secondary server to be in, and determine how often the transaction log will be restored.

As explained in the beginning, choosing the “Standby” mode will allow users to access the secondary DB. This may prevent the log from being restored and fail the log shipping process. When choosing this mode, it’s highly recommended to enable the “Disconnect users” option.

Step 3 – Monitor Server (Optional):

After configuring both servers, you can configure a third, monitor server.

The monitor server keeps track of when the transaction log on the primary database was last backed up, when the secondary servers last copied and restored the backup files, and information about any backup failure alerts.

A single monitor server can monitor multiple log shipping configurations.

You can also generate a script for all of the configurations made to reuse on other servers.

That’s it!!

You’re all done. After pressing “OK” you should see the following window, and the log shipping is up and running.

Created objects:

You might have guessed that the log shipping solution is actually implemented using a series of jobs.

On the primary server you’ll find a job responsible for backing up the log file as specified in the first step.

On the secondary server you’ll find two jobs. The first is responsible for copying the file from the primary to the secondary server, and the second is responsible for restoring the log file as defined in the second step.

On the monitor server you’ll find a job responsible for alerting when no restore has occurred.

You should also find the original log backups on the folder you specified on the primary server, and the copied ones on the secondary.

Fail Over:

There’s no automatic fail over when using log shipping.

Performing the manual fail over involves two simple steps

1) Restoring the secondary server with recovery

2) Disable all the related jobs

Pros and Cons:

I'll start with the good stuff. It’s easy! The default configuration shouldn’t take more than a few minutes to set up. In addition, the standby server can be used for reporting purposes and can be configured on the same instance.

The bad news is the frequent backups add overhead to SQL Server’s operation, and the copying of the backup files takes its toll on the disk subsystem. Another drawback is that fail over must be done manually. If an automatic fail over is important, mirroring might be a more suitable solution.

]]> http://www.madeira.co.il/log-shipping-%e2%80%93-as-easy-as-1-2-3/feed/ 1 http://www.madeira.co.il/how-are-vlfs-created-truncated-and-deleted/ http://www.madeira.co.il/how-are-vlfs-created-truncated-and-deleted/#comments Mon, 16 Jan 2012 18:14:28 +0000 ‫Shay Attiya‬ http://www.madeira.co.il/?p=3644

Today I chose to talk about Virtual Log Files or VLF in short. VLFs are the physical files that implement the log file. The log file’s log records are stored sequentially in a set of these physical files.

Creating a VLF:

Every time space is allocated for the log file (Initial creation or log growth) new VLFs are created behind the scenes. The number of new VLFs is determined by the amount of space allocated.

1) 0MB > Space Added >= 64MB ,  4 new VLFs

2) 64MB > Space Added >= 1GB ,  8 new VLFs

3) 1GB > Space Added,                 16 new VLFs

To demonstrate, I’ll use this script to create a DB with a log file in various sizes and auto growths. The database’s data file sizes is irrelevant.

USE
    master;
GO

IF DB_ID('MyDB') IS NOT NULL
BEGIN
    DROP DATABASE MyDB;
END;

CREATE DATABASE MyDB
ON (
    NAME = MyDB,
    FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQLSERVERR2_SHAY\MSSQL\DATA\MyDB.mdf',
    SIZE = 10MB,
    FILEGROWTH = 1MB)
LOG ON (
    NAME = MyDB_Log,
    FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQLSERVERR2_SHAY\MSSQL\DATA\MyDB_Log.ldf',
    SIZE = 10MB, -- 10MB, 250MB, 65MB, 1025MB
    FILEGROWTH = 6MB);
GO

To view the size and number of the VLFs created I’ll use sys.database_files and DBCC LOGINFO.

USE
    MyDB;
GO

SELECT
    File_Name    = name ,
    Size        = CAST(size*8/1024 AS VARCHAR(10))+'MB' , -- size is in pages (8KB)
    Auto_Growth    = CASE is_percent_growth
                    WHEN 1 THEN CAST(growth AS VARCHAR(10))+'%'
                    ELSE CAST(growth*8/1024 AS VARCHAR(10))+'MB'
                END
FROM sys.database_files
WHERE type_desc = 'LOG';

DBCC LOGINFO;
GO

10MB:

Our log file is 10MB, and as expected, we have 4 VLF files. But if we sum the size of the files we’ll get to a total of 10,477,568 Bytes while 10MB are 10,485,760 Bytes. There are 8,192 Bytes (size of a page) missing. This space is used by the file header and it does not store log records. You can see it in the StartOffset for the first VLF. This is true for all log file sizes.

Here are the results for bigger log files:

65MB:

250MB:

1025MB:

Ok, so now we know how VLFs are created, but before we continue, here’s a short overview on DBCC LOGINFO.

DBCC LOGINFO:

The fields that will be relevant to better understand the VLF behavior are:

FileId – If you have one log file, this number will be the same

FileSize – This number is in Bytes

StartOffset – This number is in Bytes

FSeqNo – The file sequence number. The VLF with the highest FSeqNo is the one where current log records are being written into. FSeqNo = 0 indicates that the file hasn’t been used yet.

Status – Two possible values: 0 – VLF is recyclable, 2 – VLF may be active (Highest FSeqNo must be active)

Parity – Two possible values: 64 and 128. Every time a VLF is reused, the parity value is switched.

CreateLSN – Indicates at what LNS (Log Sequence Number) the VLF was created. CreateLSN = 0 states that the VLF was created at the time the log was created. Files with the same CreateLSN were created at the same time.

To see how the VLFs are being written on, I’ll create some activity in the DB.

The DB is set for simple recovery mode for now.

USE
    MyDB;
GO

ALTER DATABASE MyDB
SET RECOVERY SIMPLE;
GO

CREATE TABLE dbo.MyTable (Value SYSNAME);

INSERT INTO dbo.MyTable
SELECT TOP 1000 T1.name
FROM sys.columns AS T1
CROSS JOIN sys.columns AS T2;
GO 20

The empty VLF files will be filled one by one until they are all full.

Notice that the first 2 VLF statuses are 0. This means that these VLFs are recyclable and can be written on.

After generating more activity the first VLF has been overwritten and its FSeqNo changed from 22 to 26. Now the 3 other VLFs are recyclable and have status = 0.

Log Truncation:

Truncation is when inactive VLFs are marked as recyclable. This is a logical operation. Truncation occurs in the following events:

1) When a checkpoint occurs (only in auto truncate mode)

2) When the log is backed up (not in COPY_ONLY)

3) When the recovery mode is set to SIMPLE

Auto Truncate Mode:

In Auto Truncate VLFs are being overwritten once they become inactive. The log file is in this mode when the DB is in SIMPLE recovery mode, or in FULL but no backup has been made.

Adding new VLFs (Growing the Log):

In order to make the log file grow and create more VLFs we must first get out if the Auto Truncate mode. To do that, we’ll set the DB to FULL recovery mode and back it up.

ALTER DATABASE MyDB
SET RECOVERY FULL;
GO

BACKUP DATABASE MyDB
TO DISK = 'C:\Users\Shay\Documents\BackUp\MyDB_BU.bak';
GO

Once we generate some more activity, all of the VLFs are full and new ones are created.

The log file grew by 6MB and 4 new VLFs are added to it. All of the files that have already been written on (FSeqNo > 0) are marked as active (Status = 2) and cannot be truncated.

Forcing Truncation:

In order to be able to recycle the VLFs, we’ll buck up the Log File.

BACKUP LOG MyDB
TO DISK = 'C:\Users\Shay\Documents\BackUp\MyDB_LOGBU.bak';
GO

After it has been bucked up, all of the VLFs are marked as inactive (Status = 0) except for the VLF that is being used at the moment. Note that the log file hasn’t shrunk, and is still 16MB.

Now new log records can be written on the truncated VLFs.

Deleting VLFs (Shrinking the Log):

If the log file is too big, it can be shrunk by removing inactive VLFs from it. To do that we’ll make the DB go into Auto Truncate mode by set the recovery mode back to SIMPLE. If we are already in SIMPLE recovery mode, we can force a truncation by using the CHECKPOINT command.

ALTER DATABASE MyDB
SET RECOVERY SIMPLE;
GO

CHECKPOINT;
GO

Now we can shrink the log file

DBCC SHRINKFILE(2,5); -- File Id, File Size in MB
GO

The file has shrunk to 6MB instead of 5 like we wanted. The reason is not all of the inactive VLFs are being removed during the file shrinking. In addition, the size of a single VLF can never change.

Now What?

Now after we have a better understanding of how VLFs are created, truncated and deleted, and have also learned how to control these events, we as DBAs can modify our database for better performance by controlling the number of VFLs in the log file.

But what do we want to achieve?

First, let’s consider our options:

Many VLFs:

Having a lot of relatively small VLFs means that the log file has more flexibility during normal activity, but may cause problems during recovery and restore, and also during CHECKPOINT when in Auto Truncate mode. All of the inactive VLFs are being inspected at these events, and having a lot of then may be time consuming.

In SQL Server 2012 an error log message will appear when going over 1,000 VLFs.

Few VLFs:

Having only a few big VLFs could cause problems involving truncation. A small amount of log records can prevent the VLF from being truncated. Shrinking the log file might suffer the same problems.

Bottom Line:

There is no conclusive best Practice when it comes to determining the desired number of VLF files in the log. DBAs who have tested the subject report the “Sweet Spot” is around 100 VLFs. Like every other aspect of DB design, you should test it on your own system and configure it for what suits you best.

If you have tested it already, I'd like it if you could share your insights.

As I’ve mentioned in my previous post, there is a catch when using a table variable as a part of a transaction. It behaves differently than a regular or a temporary table. To see this behavior, just run the attached script, once with @Rollback = 0 and once with @Rollback = 1.

SET NOCOUNT ON;

-- RollBack or not ?
DECLARE @RollBack BIT = 0; -- 1

DECLARE @Table TABLE
    (
        Id        INT ,
        Value    VARCHAR(15)
    );

INSERT INTO
    @Table
VALUES
    (1,'Update Me') ,
    (2,'Delete Me');

SELECT *
INTO #Table
FROM @Table;

SELECT
    TmpTable = T.Value ,
    VarTable = V.Value
FROM
    #Table AS T
FULL OUTER JOIN
    @Table AS V
    ON T.Id = V.Id;

SELECT Action = 'Begin Tran';

BEGIN TRAN

-- Temp table DML
INSERT INTO
    #Table
VALUES (3,'I was inserted');

UPDATE #Table
SET Value = 'I was updated'
WHERE Value = 'Update Me';

DELETE #Table
WHERE Value = 'Delete Me';

-- Table variable DML
INSERT INTO
    @Table
VALUES (3,'I was inserted');

UPDATE @Table
SET Value = 'I was updated'
WHERE Value = 'Update Me';

DELETE @Table
WHERE Value = 'Delete Me';

SELECT
    TmpTable = T.Value ,
    VarTable = V.Value
FROM
    #Table AS T
FULL OUTER JOIN
    @Table AS V
    ON T.Id = V.Id;

-- RollBack ??
IF @RollBack = 1
BEGIN
    SELECT Action = 'Rollback Tran';
    ROLLBACK TRAN;
END;
ELSE
BEGIN
    SELECT Action = 'Commit Tran';
    COMMIT TRAN;
END;

SELECT
    TmpTable = T.Value ,
    VarTable = V.Value
FROM
    #Table AS T
FULL OUTER JOIN
    @Table AS V
    ON T.Id = V.Id;

DROP TABLE #Table;

GO

The script is pretty simple:

1. Inserts identical data to a temp table and to a table variable

2. Opens a transaction

3. Perform identical DML commands (insert, update, delete) on both tables

4. Commit / Rollback the transaction

You should get the following results:

When we commit the transaction, both tables behave as expected. But when we rollback the transaction, the temp table rolls back while the table variable doesn’t.

Why is that??

The answer lies in the transaction log. Well… Actually it doesn’t, and that’s the whole point.

Every time an object in the DB changes schema or data is being modified, the change is logged in the transactions log. But a table variable is not an object, it’s a variable, and the record set it holds is not data, it’s a value. Being as such, any change to a table variable is never logged in the transaction log. In other words, SQL Server never keeps any record on table variable history, and if no history records exist, no rollback can be performed.

Be sure to keep that in mind every time you use a table variable, and there is a chance a rollback might happen.

A lot of times, when creating a procedure, we need to use some kind of an empty table in order to perform all kinds of calculations. The table is only needed for the calculation process itself, and has no other functionality otherwise. In most cases I choose to use a temporary table in order to do this, but I must admit that I’ve never given it much thought in terms of performance. It always seemed like the natural thing to do, but is it the best thing to do?

First, let consider our options:

1. Staging table (dbo.MyTable)

2.Temp table (#MyTable)

3.Table variable (@MyTable)

In order to find out which one is the best choice, I ran a little test. I used all 3 types of tables in order to execute the same kind of procedure and measured the time it took to execute. I did this for 2 kinds of procedures (simple and complex calculation) and for various record sets from 10 records to 1,000,000 records. I’ve also cleaned the cache in order to make the procedures recompile and see how it affected performance for every type of table.

The test:

I first created some test data to be used throughout the test. The tables held 1,000,000 records of random values from 1 to 10,000. The table was indexed using a clustered primary key.

-- Test data table
CREATE TABLE
    dbo.DataTable
(
    Id    INT    NOT NULL IDENTITY (1,1) ,
    Value INT    NOT NULL ,

    CONSTRAINT pk_DataTable PRIMARY KEY CLUSTERED (Id)
);

-- 1,000,000 recordes of test data
INSERT INTO
    dbo.DataTable
(
    Value
)
SELECT TOP 1000000
    ABS(CHECKSUM(NEWID())%10000) + 1
FROM
    sys.all_columns AS T1
CROSS JOIN
    sys.all_columns AS T2;
GO

I then created the Staging table. One column, no indexes, no constraints.

-- The Staging table
CREATE TABLE
    dbo.StgTable
(
    Value    INT    NOT NULL
);
GO

For the test itself I used three procedures, all doing exactly the same:

1. Populating the Staging / Temp / Variable table with a certain amount of records.

2. For the simple procedure – Count all of the values that are divided by 10.

3. For the complex procedure – Join with the DataTable and then count the data values that are divided by 10.

Staging Table:

CREATE PROCEDURE
    dbo.usp_StgTable
        @Top        INT ,
        @Complex    BIT
AS
BEGIN
    -- Insert X amount of records into the table
    INSERT INTO dbo.StgTable
    (
        Value
    )
    SELECT Value
    FROM dbo.DataTable
    WHERE Id <= @Top;

    IF @Complex = 0 -- Simple
    BEGIN
        SELECT COUNT(*)
        FROM dbo.StgTable
        WHERE Value%10 = 0;
    END;
    ELSE -- Complex
    BEGIN
        SELECT COUNT(*)
        FROM dbo.StgTable AS Stg
        INNER JOIN dbo.DataTable AS Tbl
        ON Stg.Value = Tbl.Id
        WHERE Tbl.Value%10 = 0;
    END;

END;
GO

Temp Table:

CREATE PROCEDURE
    dbo.usp_TmpTable
        @Top        INT ,
        @Complex    BIT

AS
BEGIN
    -- Insert X amount of records into the table
    SELECT Value
    INTO #Table
    FROM dbo.DataTable
    WHERE Id <= @Top;

    IF @Complex = 0 -- Simple
    BEGIN
        SELECT COUNT(*)
        FROM #Table
        WHERE Value%10 = 0;
    END;
    ELSE -- Complex
    BEGIN
        SELECT COUNT(*)
        FROM #Table AS Tmp
        INNER JOIN dbo.DataTable AS Tbl
        ON Tmp.Value = Tbl.Id
        WHERE Tbl.Value%10 = 0;
    END;

    DROP TABLE #Table;

END;
GO

Table Variable:

CREATE PROCEDURE
    dbo.usp_VarTable
        @Top        INT ,
        @Complex    BIT
AS
BEGIN
    -- Insert X amount of records into the table
    DECLARE @Table TABLE
    (
        Value    INT    NOT NULL
    );

    INSERT INTO @Table
    (
        Value
    )
    SELECT Value
    FROM dbo.DataTable
    WHERE Id <= @Top;

    IF @Complex = 0 -- Simple
    BEGIN
        SELECT COUNT(*)
        FROM @Table
        WHERE Value%10 = 0;
    END;
    ELSE -- Complex
    BEGIN
        SELECT COUNT(*)
        FROM @Table AS Var
        INNER JOIN dbo.DataTable AS Tbl
        ON Var.Value = Tbl.Id
        WHERE Tbl.Value%10 = 0;
    END;

END;
GO

I used the following scripts to run the test and to create the table where the results were saved.

CREATE TABLE
    dbo.TableTypeTest
(
    TableType     VARCHAR(10) NOT NULL ,
    Complex       BIT         NOT NULL ,
    RowCnt        INT         NOT NULL ,
    UnCompiledRun INT         NOT NULL ,
    CompiledRun   INT         NOT NULL
);
GO

The script was run twice for every table type, once for simple and the other for complex (six times in all). The script calculates the average time it took for the procedure to execute (average over 20 runs per procedure). This was done twice every cycle, once when the cache is freed before every run to make the procedure recompile every time, and the other when the execution plan is in cache.

The number of records populating the table was multiplied by 10 after every cycle (10, 100, 1000,…, 1000000) .

DBCC FREEPROCCACHE;

DECLARE
    @Top        INT = 10 ,
    @Complex    BIT = 0 , -- 1
    @Type       VARCHAR(10) = 'Stg' , -- 'Tmp', 'Var'
    @SQL        NVARCHAR(MAX);

WHILE @Top <= 1000000
BEGIN

    SET @SQL = '
    DECLARE
    @i        INT ,
    @n        INT = 20 ,
    @Time0    DATETIME2 ,
    @Time1    DATETIME2 ,
    @UnCompTime  INT = 0 ,
    @CompTime    INT = 0;

    SET @i = 1;

    WHILE @i <= @n
    BEGIN
        TRUNCATE TABLE dbo.StgTable;
        DBCC FREEPROCCACHE;
        SET @Time0 = SYSDATETIME();
        EXECUTE dbo.usp_'+@Type+'Table
            '+CAST(@Top AS VARCHAR(10))+' ,
            '+CAST(@Complex AS VARCHAR(10))+';
        SET @Time1 = SYSDATETIME();
        SET @UnCompTime += DATEDIFF(ms,@Time0,@Time1)
        SET @i += 1;
    END;

    SET @i = 1;

    WHILE @i <= @n
    BEGIN
        TRUNCATE TABLE dbo.StgTable;
        SET @Time0 = SYSDATETIME();
        EXECUTE dbo.usp_'+@Type+'Table
            '+CAST(@Top AS VARCHAR(10))+' ,
            '+CAST(@Complex AS VARCHAR(10))+';
        SET @Time1 = SYSDATETIME();
        SET @CompTime += DATEDIFF(ms,@Time0,@Time1)
        SET @i += 1;
    END;

    INSERT INTO
        dbo.TableTypeTest
    (
        TableType ,
        Complex ,
        RowCnt ,
        UnCompiledRun ,
        PreCompiledRun
    )
    SELECT
        TableType      = '''+@Type+''' ,
        Complex        = '+CAST(@Complex AS VARCHAR(10))+' ,
        RowCnt         = '+CAST(@Top AS VARCHAR(10))+' ,
        UnCompiledRun  = @UnCompTime/@n ,
        PreCompiledRun = @CompTime/@n;'

    EXECUTE sp_executesql @SQL;

    SET @Top *= 10;

END;
GO

The Results:

The “simple” procedure produced the following results (run time is shown in ms):

It’s clear that the variable table is the “winner” for the UnCompiled runs. It’s about twice to three times as fast as the Staging and Temp table. For the PreCompiled runs the Var table has a slight advantage over the Temp table, but it’s still about twice to three times as fast as the Staging table.

The “complex” procedure produced the following results:

Now the winner depends on the type of procedure (simple or complex).

For the UnCompiled runs the Var table is still 1.5-2 times faster than the Staging and Temp tables. But for the PreCompiled runs the Temp table takes the lead and is about 3 times as fast as the Var and Staging tables.

So what’s going on?

Let’s have a closer look at the results and try to find out.

We’ll start with the simple procedure.

In this procedure we count how many values are divided by 10. The execution plan for this kind of query would be the same (Table scan) no matter how many records the table holds.



Since the plan is always the same, statistics have no meaning in this case and updating it would be a useless overhead.

This is where the table variable has an advantage over other kinds of tables. It has no statistics, and the overhead for this kind of execution is kept minimal. In addition, SQL Server cannot make an estimated plan to the Temp table procedure, but is able to do so for the Var table. It doesn’t have to “wait” for the statistics to be updated for the Var table (there will never be any), but it needs to do so for the Temp table. When a Temp table is part of the procedure, the plan is recompiled every time the procedure is executed. Again, a useless overhead.

Now for the more complex procedure.

Here we join our table with another big table (1,000,000 records), before the count is performed. Now statistics have a huge influence on the execution plan. For a small amount of records, SQL Server chooses to perform the join using the “Nested Loops” method. However, for a large record count it chooses the “Hash Match” method.

When there are no statistics (Var table) SQL Server is “guessing” when it comes to choosing the right execution plan. For lower record counts, it guesses correctly having expecting only 1 record to return. For higher record counts its guess is becoming more and more incorrect.

This is why the Temp table procedure is more or less the same as the Var table procedure up to 100,000 records, but is much more efficient for higher record counts.

Here are the execution plans for 100,000 records:

Temp table:

Variable table:

This is true for both UnCompiled and PreCompiled runs, but the overhead of recompiling is still great for the Staging and Temp tables.

Conclusion

So… which is best?

The Staging table is clearly not the best choice. Having to keep statistics updated all the time is too big of an overhead for every kind of record count and every kind of procedure. In addition, there is the cost of truncating the table before (or after) every calculation. This was not measured in my little test to make the procedure as identical as possible. Naturally, this will make using a Staging table even more inefficient.

The Var table seems to be the best choice for simple calculation when there is only one way to execute the procedure. It is also the best choice when for some reason, there is no execution plan in the cache, but this is a rare event for most procedures.

The Temp table is the best choice when the procedure can be executed in more than one way. Here, updating the statistics is an overhead worth paying.

Bottom Line

As a “rule of thumb” I would use the Temp table. Usually we already have the plan in cache, so we don’t have to worry about recompilation. The advantage the Var table has over the Temp table when it comes to simple calculations is minor compared to the huge advantage the Temp table has for complex ones.

Only if you are sure SQL will choose the right execution plan, use the Var table. In any other case a Temp table is preferable.

The “Catch”

When using a table variable inside a transaction, its behavior is different than a regular or a temporary table.

Curious??

Wait for my next post…