Data Cleaning Package Sample


Updated: 5 December 2005
The Data Cleaning sample is a package that cleans data. The package uses data that is a list of names and addresses that represent potential customers. The data requires cleaning; it contains spelling errors, is missing information, and includes customers already in the database, incorrect customers, or multiple subtly different instances of the same customer.
The package control flow consists of two tasks. The first is an Execute SQL task that creates the input table, CustomerLeads, and creates the three output tables named ExistingCustomerLeads, NewCustomerLeads, and DuplicateCustomerLeads. The second task is a Data Flow data flow that executes the data flow that performs the cleaning of data extracted from the CustomerLeads table. The data flow identifies unique new, existing, and duplicate customers, and writes the rows of each customer type to the appropriate output table.
If you run the sample on a non-English version of Windows, you may have to substitute the localized name of the Program Files folder to open or run the sample.

Note:
This sample uses the Fuzzy Grouping and Fuzzy Lookup transformations, which are available only in the Enterprise version of SQL Server 2005.




Important:
Samples are provided for educational purposes only. They are not intended to be used in a production environment and have not been tested in a production environment. Microsoft does not provide technical support for these samples.



To learn more about data cleaning, search for the following articles in the MSDN Library at http://msdn.microsoft.com/library.
  • Data Cleansing Applications with SQL Server Integration Services (Windows Media Video)
  • Data Cleaning using the Fuzzy Grouping and Fuzzy Lookup Transformations (white paper)

Requirements

Running this sample package requires the following:
  • You must have installed and have administrative permissions on the AdventureWorks database.
  • If you intend only to run the sample package from the command line, you must install SQL Server 2005 Integration Services (SSIS).
  • If you intend to open the package in SSIS Designer and run the sample package, you must install Business Intelligence Development Studio. For more information about how to install samples, see "Installing Sample Integration Services Packages" in SQL Server Books Online. To obtain the latest version of the samples, including new samples released since the original release of SQL Server 2005, see SQL Server 2005 Samples and Sample Databases (April 2006).

Location of the Sample Package

If the samples were installed to the default installation location, the Data Cleaning package is located in the following folder:
C:\Program Files\Microsoft SQL Server\90\Samples\Integration Services\Package Samples\DataCleaning Sample\Data Cleaning\.
The following files are required to run this sample package.

File Description
DataCleaning.dtsx The sample package.
CreateTables.sql SQL statements to create tables.


Adding Data Viewers to the Sample

To better understand how the Data Cleaning package works, you can add data viewers to the data flow and then view the data as it moves between data flow components. We recommend that you add data viewers to the following paths:
  • Pathfrom* Union All* to OLE DB Destination-Existing Customers
  • Pathfrom* Conditional Split on Canonical Record for Group* to OLE DB Destination-Unique Customer Leads
  • Pathfrom* Conditional Split on Canonical Record for Group* to OLE DB Destination-Duplicate Customer Leads
To add data viewers
  1. Right-click the path and then click Data Viewers.
  2. In the Data Flow Path Editor, click Add.
  3. In the Configure Data Viewer dialog box, click Grid in the type list. By default, all columns display in the data viewer.
  4. Repeat steps 1-3 for other paths.

Running the Sample

The package can be run from the command line by using the dtexec utility, or can be run in Business Intelligence Development Studio.
If you are using a non-English version of Windows, you may have to update the ConnectionString property of any file connection managers used in the package to run the sample package successfully. Verify that the path used in the connection manager is valid on your computer, and if required, modify the path so that it uses the localized name of the Program Files folder.
For this sample, you may have to update "Program Files" in the ConnectionString property for the CreateTables.sql connection manager.
To run the package by using dtexec
  1. Open a Command Prompt window.
  2. Change the directory to C:\Program Files\Microsoft SQL Server\90\DTS\Binn, the location of dtexec.
  3. Type the following command: * dtexec /f "C:\Program Files\Microsoft SQL Server\90\Samples\Integration Services\Package Samples\Data Cleaning Sample\DataCleaning\DataCleaning.dtsx" *
  4. Press Enter.For more information about how to run the package by using the dtexec utility, see the topic, "dtexec Utility", in SQL Server 2005 Books Online.
To run the package in Business Intelligence Development Studio
  1. Open Business Intelligence Development Studio.
  2. On the File menu, point to Open, and then click Project/Solution.
  3. Locate the DataCleaning Sample folder, and then double-click the file named DataCleaning.sln.
  4. In Solution Explorer, right-click DataCleaning.dtsx in the SSIS Packages folder, and then click Execute Package.
Note:
If you open the package in SSIS Designer and view the package properties, you will notice that the DelayValidation property is set to True. Validation of the package must be delayed because some tables used by the Data Cleaning sample package—the CustomerLeads, and the three output tables named ExistingCustomerLeads, NewCustomerLeads, and DuplicateCustomerLeads—are not created until the first time the package runs. If DelayValidation is set to False, a validation error occurs when you open the package in SSIS Designer before running the package.



Components in Sample

The following table lists the tasks, containers, data sources and destinations, and transformations that are used within the sample.

Element Purpose
Execute SQL task The Execute SQL task is named Create Customer Address Reference Table View, Populate NewCustomer Input Table and Create Output Tables. This task creates the input table, CustomerLeads, and also creates the three output tables named ExistingCustomerLeads, NewCustomerLeads, and DuplicateCustomerLeads.
Data Flow task The Data Flow task, Fuzzy Lookup Data Flow Task, executes the data flow in the package.
OLE DB source The OLE DB source, OLE DB Source - Customer Leads, reads records from the CustomerLeads table.
Lookup transformation The Lookup transformation, Lookup against Existing Customers, performs an exact lookup to identify existing customers. If the lookup succeeds, the record is inserted into the ExistingCustomerLeads table.
Derived Column transformation The Derived Column transformation, Derived Column, adds the _Similarity columns to each row and sets the column value to 1.
Fuzzy Lookup transformation The Fuzzy Lookup transformation, Fuzzy Lookup against Existing Customers, performs a fuzzy lookup to identify customer records that are fuzzy matches of existing customer records. The transformation adds a _Similarity column that contains a similarity score to each row. The score 0.0 means no match was found, whereas 1.0 means an exact match was found. A score between 0.0 and 1.0 is a measure of similarity in which a value closer to 1.0 indicates greater similarity.
Conditional Split transformation The first Conditional Split transformation, ConditionalSplit on Similarity, directs input rows to one of two outputs depending on the value of the similarity score determined by the fuzzy lookup. Rows with a similarity score >= .70 are written to the ExistingCustomerLeads table. Rows with similarity scores < 70 are probably valid new customer leads and additional cleaning is done on these rows. The second Conditional Split transformation, Conditional Split on Canonical Record for Group, directs input rows to one of two outputs depending on whether the data row is a duplicate. If the values of the keyin and keyout columns are equal, the row is used as the canonical row in the group, and the canonical row is inserted into the NewCustomerLeads table. If the keyin and key_out columns are not equal, the row is treated as a fuzzy duplicate and the row is inserted into the DuplicateCustomerLeads table.
Union All transformation The Union All transformation, Union All, merges rows of existing customers—both exact and fuzzy matches—into one dataset.
Fuzzy Grouping transformation The Fuzzy Grouping transformation, Fuzzy Grouping, groups customers who are likely duplicates. The transformation adds three columns keyin, keyout and score to each row. keyin is a unique identifier assigned to each input row and keyout contains the particular keyin assigned to the row that best represents all the rows in a fuzzy group. All rows in a fuzzy group will have the same keyout value. The score column is a value between 0.0 and 1.0 that describes the textual similarity between a given input row and the row selected to be the canonical value.
OLE DB destinations The OLE DB destination, OLE DB Destination - Existing Customers, inserts rows into the ExistingCustomerLeads table.The OLE DB destination, OLE DB Destination - Unique Customer Leads, inserts rows into the NewCustomerLeads table.The OLE DB destination, OLE DB Destination - Duplicate Customer Leads, inserts rows into the DuplicateCustomerLeads table.
File connection manager The File connection manager, CreateTables.sql, connects to the file that contains the SQL the package uses.
OLE DB connection manager The OLE DB connection manager, (local).AdventureWorks, connects to the AdventureWorks database on the local server.


The following table describes the data in the output tables.

Table Description
ExistingCustomerLeads Contains records that exactly match an existing customer, and records that fuzzily match an existing customer with very high textual similarity.
NewCustomerLeads Contains records for which there was no good match to an existing customer. If the list contained multiple instances of the same name, or a highly similar version of a particular name, only one record will be directed to NewCustomerLeads, and the duplicates will be directed to DuplicateCustomerLeads.
DuplicateCustomerLeads Contains duplicates of new customers.


Sample Results

To see the execution results of the Data Cleaning sample package, run the following Transact-SQL query:

Select * from AdventureWorks.FuzzyLookupExample.ExistingCustomerLeads
Select * from AdventureWorks.FuzzyLookupExample.NewCustomerLeads
Select * from AdventureWorks.FuzzyLookupExample.DuplicateCustomerLeads |

Change History


Release History
5 December 2005 Changed content: Corrected the schema name to use in the SELECT statements that return execution results.


© 2007 Microsoft Corporation. All rights reserved.

Last edited Nov 7, 2008 at 12:02 AM by bonniefe, version 15

Comments

Ashwaniroy Feb 10, 2009 at 4:17 AM 
I used the MSI installer and I was able to get the SP mentioned above. There is no issue with the sample.

USE [AdventureWorks]
GO
/****** Object: StoredProcedure [dbo].[CreateTables] Script Date: 02/10/2009 04:06:19 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[CreateTables]
AS
BEGIN
BEGIN TRY
IF NOT EXISTS(SELECT * FROM sys.schemas WHERE name='FuzzyLookupExample') EXECUTE('CREATE SCHEMA FuzzyLookupExample')

IF NOT EXISTS(SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='FuzzyLookupExample' AND TABLE_NAME = 'vCustomerAddress')
EXECUTE('
CREATE VIEW FuzzyLookupExample.vCustomerAddress AS
SELECT I.CustomerID, C.FirstName, C.LastName, A.AddressLine1, A.City,
SP.Name AS State, CR.Name AS CountryRegion
FROM Person.Contact AS C
JOIN Sales.Individual AS I ON C.ContactID = I.ContactID
JOIN Sales.CustomerAddress AS CA ON CA.CustomerID = I.CustomerID
JOIN Person.Address AS A ON A.AddressID = CA.AddressID
JOIN Person.StateProvince SP ON SP.StateProvinceID = A.StateProvinceID
JOIN Person.CountryRegion CR ON CR.CountryRegionCode = SP.CountryRegionCode
')

IF EXISTS(SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='FuzzyLookupExample' AND TABLE_NAME = 'CustomerLeads') DROP TABLE FuzzyLookupExample.CustomerLeads

CREATE TABLE FuzzyLookupExample.CustomerLeads (FirstName nvarchar(50), LastName nvarchar(50), AddressLine1 nvarchar(60), City nvarchar(30), State nvarchar(50), CountryRegion nvarchar(50))
INSERT INTO FuzzyLookupExample.CustomerLeads values ('Jon', 'Yang', '3761 N. 14th St', 'Rockhampton', 'Queensland', 'Australia')
INSERT INTO FuzzyLookupExample.CustomerLeads values ('E.', 'Huang', '2243 W St', 'Seaford', 'Victoria', null)
INSERT INTO FuzzyLookupExample.CustomerLeads values ('Reuben', 'Tores', '5844 Linden', 'Hobart', null, null)
INSERT INTO FuzzyLookupExample.CustomerLeads values ('Chris', 'Z.', '1825 Village Place', 'NR', 'NS Wales', 'Australia')

INSERT INTO FuzzyLookupExample.CustomerLeads values ('Pannarat', 'Pattanapitakkul', '1 Microsoft Way', 'Redmond', 'Washington', 'USA')
INSERT INTO FuzzyLookupExample.CustomerLeads values ('pannarat', 'pattanapitakkul', 'Microsoft Way', 'Redmond', 'Washington', 'USA')
INSERT INTO FuzzyLookupExample.CustomerLeads values ('Pannarat', 'Paddanapitakul', '1 Microsoft Way', 'Redmond', 'Washington', null)
INSERT INTO FuzzyLookupExample.CustomerLeads values ('Pan', 'Pattanapitakkul', '1 Microsoft Way', 'Redmond', null, 'USA')
INSERT INTO FuzzyLookupExample.CustomerLeads values ('pannarat', 'pattanapitakkul', 'One microsoft way', 'redmond', 'washington', 'u.s.a.')
INSERT INTO FuzzyLookupExample.CustomerLeads values ('Panarat', 'Pattanapittakul', 'One Microsoft Wy.', 'Redmond', 'Wash', 'USA')

INSERT INTO FuzzyLookupExample.CustomerLeads values (N'Göktuğ', N'Oğuz', '111 Broadway Avenue', 'Seattle', 'Washington', 'USA')
INSERT INTO FuzzyLookupExample.CustomerLeads values (N'Gökktuğ', N'Oğuz', '111 Broadway', 'Seattle', 'WA', null)
INSERT INTO FuzzyLookupExample.CustomerLeads values (N'goktug', N'ogguz', '111 broadway ave', 'seattle', null, null)
INSERT INTO FuzzyLookupExample.CustomerLeads values (N'Goktuğ', N'Oğuz', '111 Broadway Ave', 'Seattle', 'WA', 'USA')
INSERT INTO FuzzyLookupExample.CustomerLeads values (N'Göktuğ', N'Oğuz', '111 Broadway Ave.', null, 'Washington', 'USA')

IF EXISTS(SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='FuzzyLookupExample' AND TABLE_NAME = 'ExistingCustomerLeads') DROP TABLE FuzzyLookupExample.ExistingCustomerLeads

CREATE TABLE FuzzyLookupExample.ExistingCustomerLeads (
[FirstName (Input)] NVARCHAR(50),
[LastName (Input)] NVARCHAR(50),
[AddressLine1 (Input)] NVARCHAR(60),
[City (Input)] NVARCHAR(30),
[State (Input)] NVARCHAR(50),
[CountryRegion (Input)] NVARCHAR(50),
[FirstName (Matched)] NVARCHAR(50),
[LastName (Matched)] NVARCHAR(50),
[AddressLine1 (Matched)] NVARCHAR(60),
[City (Matched)] NVARCHAR(30),
[State (Matched)] NVARCHAR(50),
[CountryRegion (Matched)] NVARCHAR(50),
[_Similarity] REAL,
[_Similarity_FirstName] REAL,
[_Similarity_LastName] REAL,
[_Similarity_AddressLine1] REAL,
[_Similarity_City] REAL,
[_Similarity_State] REAL,
[_Similarity_CountryRegion] REAL
)

IF EXISTS(SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='FuzzyLookupExample' AND TABLE_NAME = 'NewCustomerLeads') DROP TABLE FuzzyLookupExample.NewCustomerLeads


CREATE TABLE FuzzyLookupExample.NewCustomerLeads (
[_key_in] INT,
[_key_out] INT,
[FirstName] NVARCHAR(50),
[LastName] NVARCHAR(50),
[AddressLine1] NVARCHAR(60),
[City] NVARCHAR(30),
[State] NVARCHAR(50),
[CountryRegion] NVARCHAR(50)
)

IF EXISTS(SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='FuzzyLookupExample' AND TABLE_NAME = 'DuplicateCustomerLeads') DROP TABLE FuzzyLookupExample.DuplicateCustomerLeads

CREATE TABLE FuzzyLookupExample.DuplicateCustomerLeads (
[_key_in] INT,
[_key_out] INT,
[FirstName] NVARCHAR(50),
[LastName] NVARCHAR(50),
[AddressLine1] NVARCHAR(60),
[City] NVARCHAR(30),
[State] NVARCHAR(50),
[CountryRegion] NVARCHAR(50),
[FirstName_clean] NVARCHAR(50),
[LastName_clean] NVARCHAR(50),
[AddressLine1_clean] NVARCHAR(60),
[City_clean] NVARCHAR(30),
[State_clean] NVARCHAR(50),
[CountryRegion_clean] NVARCHAR(50),
[_Similarity] REAL,
[_Similarity_FirstName] REAL,
[_Similarity_LastName] REAL,
[_Similarity_AddressLine1] REAL,
[_Similarity_City] REAL,
[_Similarity_State] REAL,
[_Similarity_CountryRegion] REAL
)
RETURN 0;
END TRY
BEGIN CATCH
RETURN 1;
END CATCH
END

Ashwaniroy Feb 10, 2009 at 3:15 AM 
I do like this package and reusability of the concept and code but the second execute SQL task Create "CustomerAddress Reference Table View, Populate NewCustomers Input Table and Create Output Tables" uses a SP Exec CreateTables. I am not able to find the SP in the sample DB or the Package.