Upload
cecil-sanders
View
213
Download
0
Embed Size (px)
Citation preview
Query DIFF Utility
Comparing the data
In 2 tables (or queries)
-- show the differences --
MDCFUG
December 10, 2002
Joan Falcã[email protected]
“Why would you want to do that?”
Specifically – verify that data transferred from one system to another is identical before deleting it at the source.
Generally – have a quality-assurance tool to analyze data differences: Verify expected matches Report on expected differences Identify surprise differences and matches
Applications Find Differences in 2 similar tables
See how a table has changed over the week QA for data entry effort (before and after) Verify expected changes due to code mods
Compare the column names from MS-SQL and Oracle schemas Or compare length of text columns
Analyze whether duplicate keys are duplicate rows (heavy hand on the “submit” button?)
See the results of modifying a query Inner join vs outer join Null predicate
More Applications: QA for Ported Systems or Regression Testing
dataconversion
routine
oldsystemupdate
newsystemupdate updated
porteddbms
dbms
dataconversion
routineupdated
dbms
porteddbms
portedupdated
dbms
Difference?
Reinventing the wheel?
http://devex.macromedia.com/developer/gallery/index.cfm Cf_Venn: Remember Venn diagrams from school? This
Custom Tag compares two lists, performing any or all of these operations: Union, Intersection, and Differences.
TDataSourceCompare: Compares all tables, columns, datatypes of 2 MS SQL database objects that are registered as datasources on your server. Used for checking your Production database and Development database to make sure your columns all match before a critical launch. DB1 First Datasource name. DB2 Second Datasource name.
CF_List_Compare is a custom tag that will compare the values of two given lists (List 1, and List 2)
Reinventing the wheel?
August 7, 2002SQL Script for Comparing the Contents of Two Tables
Here's some SQL code from Eli Leiba that will all you to compare two tables -- say, table A and table B -- to determine if their content is the same. Assuming that A and B have the same structure, here's how it works. First, from set theory, recall that:
If ((|A| = |B|) && |A U B| = |A|)) ====>>> A = B |A| = NUMBER of rows in A |B| = NUMBER of rows in B
http://www.databasejournal.com/features/mssql/article.php/1441271
Reinventing the wheel?
http://www.databasejournal.com/features/mssql/article.php/1441271(continued)
Here's the SQL code
(with T-SQL syntax):
declare @cnt1 int
declare @cnt2 int
declare @cnt3 int
declare @res bit
select @cnt1 = count(*) from A
select @cnt2 = count(*) from B
select @cnt3 = count('x')
from (select * from A
UNION
select * from B) as t
if (@cnt1 = @cnt2)
and (@cnt2 = @cnt3)
begin
set @res = 1
print 'A = B'
end
else
begin
set @res = 0
print 'A <> B'
end
go
Previous implementation – UNION/JOIN query in Access
Query is messy: query for each column name query for extra rows in each of the 2 tables extra predicates for nulls extra predicates to flag duplicate rows
(should table not have a primary key)
Access permits union across dissimilar columns (different data types)
Access DOES permit inner join and union across linked tables from different DBMS, but I have not tested this.
Previous implementation – UNIVAC’s Symbolic Stream Generator
Generates source code (to be compiled and executed) according to parameters
Old technology (no longer available) Required a lot of parameters
One spec per column (location and type)
Compared 2 files – very limited A query is much more flexible
General ColdFusion Approach
Define 2 queries With the same column names
rename with AS, if needed With a common key (possibly multi-column) Sort on the common key
Merge queries on the common key Report extra keys Report data differences when keys match
Feedback for user Column names Notification of errors
Parameters in General-Purpose Differencing
In a FORM, prompt for: Datasource Table specification (query) Key specification Sort order Login parameters (username/password)
Demos
Challenges in General-Purpose Programming
Column name(s) not known for SELECT *
Number of columns not known Number of key columns not known Column name syntax in ORDER BY
clause differs across databases
ColdFusion essential features
CFQUERY’s returned variable: queryname.ColumnList
ListGetAt(ISQLquery1.ColumnList,ii)
CFQUERY column indexing: queryname.columnname[index]
Evalute function:Evaluate(‘queryname.columnname’)
Evaluate("ISQLquery1."&ListGetAt(ISQLquery1,ii)&"["&q1&"]")
Unexpected problems False matches
SQL Case-sensitive vs CF case-insensitive ColdFusion null same as empty string
(unless query uses COALESCE to give a value to nulls)
False differences Extra rows on both sides (“ships passing in the night”) Trailing blanks (do not count in SQL, do in CF)
Sort sequences differ Across database products Across database configurations of same product ColdFusion has its own sort sequence
SQL query special characters hard to store need URLEncodedFormat
MoreDemos
Workarounds to sort problems
Sort by UPPER(columnname) Replace known special characters with
something that sorts the same UPPER(REPLACE(columnname,’_’,’ ’))
ReSort in ColdFusion, via CF5 query-on-query
Additional Problems In CFFORM my own javascript
inadvertantly disables CFINPUT validation onSubmit="return SubmitDiffForm(_CF_this);“
Column names with special characters won’t “evaluate” (I warn and abort)
My javascript function to uppercase character sort columns was too confusing (now disabled)
Additional Implementation Notes <CFSETTING EnableCFOUTPUTONLY="Yes">
To limit extraneous white space (lots of it) Hidden radio button trick – form variable is always
present and has values "Y,N“ or “N” thisTag.generatedContent gives “playback” Empty username/password attributes mean “use
defaults” (instead of ODBC error) PreserveSingleQuotes needed in CFQUERY Sandbox security may limit DIFFing Written/tested in CF 4.5; enhanced with optional
CF5 features
Security concerns This utility must be protected;
Otherwise, it’s a hacker’s dream NOTE: I use a session variable to hold a login
Security can be disabled (in application.cfm, set BypassSecurity=“Yes”) OK, iff protection already exists
Security must be configured Use an existing username/password to login
Avoids password management Provide parameters in LoginParam.cfm
Database passwords for queries are omitted in stored DIFF specification
Feature Creep (extra funcionality) Javascript function cascades merge key
to sort key input boxes (less data entry) Option to save entire DIFF specification
in a file Batch execution (can be scheduled in CF admin) Email the differences report Resort (CF5 query-on-query) to avoid sort
sequence variance
Future Enhancements?
Case-sensitive search Parameters to limit output Numeric tolerances in a match Encrypted passwords for datasources CfScript to speed it up