SQL: CHANGE TRACKING FOR ENTITY

sql entity tracking trace

186 просмотра

1 ответ

68 Репутация автора

QUESTION:

What approach should I use to notify one databases about the changes made to a table in another database. Note: I need one notification per statement level event, this includes the merge statement which does a insert, update and delete in one.

BACKGROUND:

We're working with a third party to transfer some data from one system to another. There are two databases of interest here, one which the third party populates with normalised staging data and a second database which will be populated with the de-normalised post processed data. I've created MERGE scripts which do the heavy lifting of processing and transferral of the data from these staging tables into our shiny denormalised version, and I've written a framework which manages the data dependencies such that look-up tables are populated prior to the main data etc.

I need a reliable way to be notified of when the staging tables are updated so that my import scripts are run autonomously.

METHODS CONSIDERED:

SQL DML Triggers

I initially created a generic trigger which sends change information to the denormalised database via service broker, however this trigger is firing three times, once for insert, update and delete and is thus sending three distinct messages which is causing the import process to run three times for a single data change. It should be noted that these staging tables are also being updated using the MERGE functionality within SQL Server, so is handled in a single statement.

SQL Query Notification

This appears to be perfect for what I need, however there doesn't appear to be anyway to subscribe to notifications from within SQL Server, this can only be of used to notify change at an application layer written in .net. I guess I maybe able to manage this via CLR integration, however I'd still need to drive the notification down to the processing database to trigger the import process. This appears to be my best option although it will be long winded, difficult to debug and monitor, and probably over complicating an otherwise simple issue.

SQL Event Notification

This would be perfect although doesn't appear to function for DML, regardless of what you might find in the MS documentation. The create event notification command takes a single parameter for event_type so can be thought of as operating at the database level. DML operates at an entity level and there doesn't appear to be anyway to target a specific entity using the defined syntax.

SQL Change Tracking

This appears to capture changes on a database table but at a row level and this seems to be too heavy handed for what I require. I simply need to know that a change has happened, I'm not really interested in which rows or how many, besides I'd still need to convert this into an event to trigger the import process.

SQL Change Data Capture

This is an extension of Change Tracking and records both the change and the history of the change at the row level. This is again far too detailed and still leaves me with the issue of turning this into a notification of some kind so that import process can be kicked off.

SQL Server Default Trace / Audit

This appears to require a target which must be of either a Windows Application / Security event log or a file on the IO which I'd struggle to monitor and hook into for changes.

ADDITIONAL

My trigger based method would work wonderfully if only the trigger was fired once. I have considered creating a table to record the first of the three DML commands which could then be used to suspend the posting of information within the other two trigger operations, however I'm reasonable sure that all three DML triggers (insert, update delete) will fire in parallel rending this method futile.

Can anyone please advise on a suitable approach that ideally doesn't use a scheduled job to check for changes. Any suggestions gratefully received.

Автор: user466015 Источник Размещён: 18.07.2016 12:01

Ответы (1)


0 плюса

68 Репутация автора

This simplest approach has been to create a secondary table to record when the trigger code is run.

CREATE TABLE [service].[SuspendTrigger]
(

    [Index]     [int] IDENTITY(1,1)     NOT NULL,
    [Name]      [nvarchar](200)         NOT NULL,
    [DateTime]  [datetime]              NOT NULL,
    [SPID]      [int]                   NOT NULL,

 CONSTRAINT [pk_suspendtrigger_index] PRIMARY KEY CLUSTERED 
(
    [Index] ASC
) ON [PRIMARY]
) ON [PRIMARY]

Triggers run sequentially so even when a merge statement is applied to an existing table the insert, update and delete trigger code run one after the other.

The first time we enter the trigger we can therefore write to this suspension table to record the event and then execute what ever code needs to be executed.

The second time we enter the trigger we can check to see if a record already exists and therefore prevent execution of any further statements.

alter trigger [dbo].[trg_ADDRESSES]
       on  [dbo].[ADDRESSES]
       after insert, update, delete
    as 
    begin
        set nocount on;

        -- determine the trigger action - not trigger may fire
        -- when nothing in either update or delete table
        ------------------------------------------------------
        declare @action as nvarchar(6) = (case  when (  exists ( select top 1 1 from inserted   )
                                                    and exists ( select top 1 1 from deleted    ))  then N'UPDATE'
                                                when    exists ( select top 1 1 from inserted   )   then N'INSERT'
                                                when    exists ( select top 1 1 from deleted    )   then N'DELETE'     
                                            end)

        -- check for valid action
        -------------------------
        if @action is not null
            begin
                if not exists ( select  * 
                            from    [service].[SuspendTrigger] as [suspend]
                            where   [suspend].[SPID] = @@SPID
                            and     [suspend].[DateTime] >= dateadd(millisecond, -300, getdate())
                        )
                begin

                    -- insert a suspension event
                    -----------------------------
                    insert into [service].[SuspendTrigger] 
                    (
                        [Name]      , 

                        [DateTime]  , 
                        [SPID]
                    )
                    select  object_name(@@procid)   as [Name]       ,
                            getdate()               as [DateTime]   ,
                            @@SPID                  as [SPID]

                    -- determine the message content to send
                    ----------------------------------------
                    declare @content xml = (

                        select  getdate()               as [datetime]                   ,
                                db_name()               as [source/catelogue]           ,
                                'DBO'                   as [source/table]               ,
                                'ADDRESS'               as [source/schema]              ,
                            (select     [sessions].[session_id]             as [@id]                        ,
                                        [sessions].[login_time]             as [login_time]                 ,
                                        case when ([sessions].[total_elapsed_time] >= 864000000000) then
                                            formatmessage('%02i DAYS %02i:%02i:%02i.%04i', 
                                                (([sessions].[total_elapsed_time]    / 10000 / 1000 / 60 / 60 / 24)),
                                                (([sessions].[total_elapsed_time]    / (1000*60*60)) % 24),
                                                (([sessions].[total_elapsed_time]    / (1000*60)) % 60),
                                                (([sessions].[total_elapsed_time]    / (1000*01)) % 60),
                                                (([sessions].[total_elapsed_time]    ) % 1000)) 
                                        else 
                                            formatmessage('%02i:%02i:%02i.%i', 
                                                (([sessions].[total_elapsed_time]    / (1000*60*60)) % 24),
                                                (([sessions].[total_elapsed_time]    / (1000*60)) % 60),
                                                (([sessions].[total_elapsed_time]    / (1000*01)) % 60),
                                                (([sessions].[total_elapsed_time]    ) % 1000))     
                                        end                                 as [duration]                   ,
                                        [sessions].[row_count]              as [row_count]                  ,
                                        [sessions].[reads]                  as [reads]                      ,   
                                        [sessions].[writes]                 as [writes]                     ,
                                        [sessions].[program_name]           as [identity/program_name]      ,
                                        [sessions].[host_name]              as [identity/host_name]         ,
                                        [sessions].[nt_user_name]           as [identity/nt_user_name]      ,   
                                        [sessions].[login_name]             as [identity/login_name]        ,
                                        [sessions].[original_login_name]    as [identity/original_name]     

                                from [sys].[dm_exec_sessions] as [sessions]
                                where [sessions].[session_id] = @@SPID
                                for xml path('session'), type) 

                            for xml path('persistence_change'), root('change_tracking'))

                            -- holds the current procedure name
                            -----------------------------------
                            declare @procedure_name nvarchar(200) = object_name(@@procid)

                            -- send a message to any remote listeners
                            -----------------------------------------
                            exec [service].[usp_post_content_via_service_broker] @MessageContentType = 'Source Data Change', @MessageContent = @content, @CallOriginator = @procedure_name

                end
            end
        end
    GO;

All we need to do now is create an index on the [datetime] field within the suspension table so that this is used during the check. I'll probably also create a job to clear down any entries older than a couple of minutes to try to keep the contents down.

Either way, this provides a way of ensuring that only on notification is generated per table level modification.


if your interested the message contents will look something like this ...

      <change_tracking>
        <persistence_change>
          <datetime>2016-08-01T16:08:10.880</datetime>
          <source>
            <catelogue>[MY DATABASE NAME]</catelogue>
            <table>DBO</table>
            <schema>ADDRESS</schema>
          </source>
          <session id="1014">
            <login_time>2016-08-01T15:03:01.993</login_time>
            <duration>00:00:01.337</duration>
            <row_count>1</row_count>
            <reads>37</reads>
            <writes>68</writes>
            <identity>
              <program_name>Microsoft SQL Server Management Studio - Query</program_name>
              <host_name>[COMPUTER NAME]</host_name>
              <nt_user_name>[MY ACCOUNT]</nt_user_name>
              <login_name>[MY DOMAIN]\[MY ACCOUNT]</login_name>
              <original_name>[MY DOMAIN]\[MY ACCOUNT]</original_name>
            </identity>
          </session>
        </persistence_change>
      </change_tracking>

I could send over the action that triggered the notification but I'm only really interested in the fact that some data has changed in this table.

Автор: user466015 Размещён: 01.08.2016 03:29
Вопросы из категории :
32x32