Upload
kenya-jeanes
View
235
Download
7
Embed Size (px)
Citation preview
ENOVIA SmarTeamData Loading Strategies
Marc Young, CEO
xLM Solutions, LLC
Strategies to ease the migration pain…Loading, or migrating, data into ENOVIA SmarTeam can be a daunting task…
With diligent planning and thorough testing, the process can go smoothly
• Data Clean-up and Process Review• Data Formats• Data Loading Rules• Migration Cost Estimation
1. Pre-Migration Data Analysis
• Bulk Loader• Import Tools• Custom Scripts
2. Data Loading Options
• Testing the Migration Process• Data Verification• Pre-Data Load Planning• Production Data Load
3. Execution
Data AnalysisAnalyze the Data to be Loaded
Determine legacy data nature and
location• File-/Folder-based• Legacy systems
(databases, vaults, etc.)
• Hard copy (e.g., aperture cards)
• Type of data: CAD (with internal links), office, etc.
Anticipate data access issues
• All sources available?
• Third-party systems accesses?
• Additional experts?• Additional
resources?
Is CAD data conversion needed?
• e.g., ProE → CATIA• V4 → V5
Data volume and estimated time
to load• Will/Can downtime
be scheduled for data loading?
Analysis: Data Clean-Up
Typically
involves
human intellige
nce
May be time consuming
May not be worthwhile
Opportunity for Data
Clean-Up
Analysis: Data Clean-Up
What types of data clean-up are required?• Duplicated part numbers per file• Duplicated file names• Missing or incorrect file attributes• Missing or incorrect file associations or links• File revision not current• Non-current parts in current assemblies• Non-compliance or old revision/versioning
scheme
Validate data accuracy
Analysis: Process Reviews and Updates
• An opportunity to review business processes
• Opportunity to evaluate and update user experiences
• Important: new changes to the underlying data model must be verified with the migration processes
Analysis: Data FormatsCAD Data• File-based• Typically has relationships among files• Attributes and properties contained within the file
Meta Data to be Loaded or Added• Text description of data external to the associated (CAD) file• Users
• Import from existing Active Directory• Lookup Lists• Projects• File location pointers
Combined Data• CAD data with embedded relationships; but…• Attributes and properties reside outside of the CAD file
• e.g., in an Excel spreadsheet• Construct a link between the CAD file and attribute data
CAD Data
Meta Data to be Loaded or Added
Combined Data
Analysis: Determine the Data Loading RulesWhat are the rules to determine
where the data goes?
How will data fields be mapped from legacy system to SmarTeam?
What state(s) will the data be loaded to?
Will the data be linked to other SmarTeam data?
How to handle the cases with missing data?
Which data revision to load: latest or full history?
• Should the data be located in a folder?• Should a folder be automatically
created for the data?
• Text, Lookup, Reference To Class, URL
• If so, what is the ruleset?
• If a full data history is desired, determine how the revision history is maintained in the legacy system
Analysis: Effort and Cost Estimations
Data Load Project Cost Quantification
Time (data loading tools development
and testing)
Price
Resources involved
Impact of downtime for
users
• Data Clean-up and Process Review• Data Formats• Data Loading Rules• Migration Cost Estimation
1. Pre-Migration Data Analysis
• Manual Processing• SmarTeam Import and Export Tools• Custom Scripts (executables, add-ins, etc)• Combined Manual Processing and Custom Scripting• Pre-Developed Data Migration Programs
2. Data Loading Options
• Testing the Migration Process• Data Verification• Pre-Data Load Planning• Production Data Load
3. Execution
2. Data Loading Options
3. Execution
Overview SmarTeam Bulk Loading Tool• Load CAD File data that
SmarTeam has integration for• CATIA, SolidWorks, SolidEdge,
Inventor, AutoCAD• Highlight a set of files that should
be loaded into SmarTeam• The bulk loader will open each file
in the CAD application one at time and perform a SmarTeam Save
• Recommended way to bulk load standard parts
Setup for SmarTeam Bulk Loading Tool• CAD Integration should be configured
• Associated CAD File Type to SmarTeam Class• Map CAD Attributes to SmarTeam field
• Sequence needs to be assigned to the primary Id• Batch Mode Save Option should be turned on
• Otherwise, you will have to hit okay for each file to be loaded.
• Default Class to be specified
Issues with SmarTeam Bulk Loading Tool• Speed – an open and save performed on all files being loaded• CAD data integrity – If links are bad/missing in the CAD data:
• the files will not be saved• may require user intervention per occurrence
• Attribute data needs to exist in CAD properties* • No log created for data load*• Difficult if CAD data is to be associated with more than one class*
• An Engineering Part, Manufacturing Part, Standard Part• All data is linked to only one project*• May require copying CAD to data from various directories to one
directory• Does not handle loading of multiple revisions
*Avoidable with scripting
Overview of SmarTeam Import Tool• Out of the box tool provided by
SmarTeam to load meta data• The ability to pick a class to load data
• Leaf Classes• Link Classes i.e. Document Project
Links• The ability to map data from:
• Database – SmarTeam DB or non SmarTeam DB
• Text File – Delimited or Fixed Width• Default Value – Defined in SmarTeam• Default Value – Specific for the data
load• Populate field by a Sequence
• Log created for data that is imported• Import Tool can run in an Add or Update
mode
Issues with SmarTeam Import Tool• Limited to 32,000 records per load
• Greater than 32,000, data needs to be split and multiple loads conducted
• Tool can be “buggy” • Crashing from time to time• Data Mappings may get lost
• Tool is manual• For each class to be imported mappings
should be configured• Can be tedious to do for many classes
• Tool does not handle certain condition (with out Scripting)• Loading multiple revisions and specifying
the latest revision• Loading data to internal SmarTeam fields or
Reference to Class fields
Overview Custom Scripts / Programs• Best suited for larger data
migrations• Almost any type of data loading
rule can be created• Determine what class to load
the data • Modify data to be loaded
based on data in the system• Re-formatted before going
into SmarTeam• Need to create or build link
relationships in the system• Various complex Data load
scenarios will be conducted• Need custom logging an error
handling
Custom Script Data Loading - Simple Example• Data exists in a legacy Folder• File Name contains meta data attributes such as• ID_Descritpion_Rev• .doc files will go into an Engineering folder• .xls files will go into an Manufacturing folder• Project will need to be specified• Maintain latest revison
SmarTeam Custom Script Tool Example
Combination Bulk Load and Custom script / Import Utility• Bulk Load CAD data into
SmarTeam• Hook scripts to the SmarTeam
“Before Add” Event to retrieve information • i.e. from the file name to
update fields on the Profile Cards
• Write custom code that will query the database based on the file name• Update information from
another source where the file name is used to link the file data /object already in SmarTeam and the other external source for data
• Data Clean-up and Process Review• Data Formats• Data Loading Rules• Migration Cost Estimation
1. Pre-Migration Data Analysis
• Bulk Loader• SmarTeam Import and Export Tools• Custom Scripts (executables, add-ins, etc)• Combined Manual Processing and Custom Scripting• Pre-Developed Data Migration Programs
2. Data Loading Options
• Testing the Migration Process• Data Verification• Pre-Data Load Planning• Production Data Load
3. Execution
2. Data Loading Options
3. Execution
Execution:Testing the Migration Process
• Test the migration process!!!• Mitigate all discovered issues
• Legacy data issues» Missing data files» Orphan links and/or blank fields» Inconsistent legacy data such as revision schema» Bad formats
– E.g., Attempts to load characters into integer fields– Class determination rules fail
» Repairing the legacy data– Manual clean-up– Change data load rules (i.e., change the code)– Modify the SmarTeam data model to better accommodate the
legacy data
• Bug(s) in the migration tool code
• Data loading procedural issues» e.g., failure to delete all files with a creation date before 12/31/1999
• Document the migration procedure or update the tool to automate the migration process
• Test again!!!
Crucial to implement a data verification strategy
Analyze migration process log files Spot-check data
Randomly verify the data objects loaded Check at least 10% of data
User validation (done through testing with the pilot system) – an important part of the verification process
User validations >> IT/CAD administrators sign-offs
Develop a tool to compare and analyze the data in SmarTeam and legacy system(s) Verify the number of records between systems Ensure pertinent fields and data are preserved
in both systems
Execution: Pre-Data Load Planning• When will the legacy system be turned off?• How long will it take to load the data?• Will the data loading occur during non-
business hours?• Specify the data load process:
o What are the required pre-data loading activities/events?
• e.g., legacy data clean-up
o Document the data load sequence• Populate the lookup tables• Import users before documents• Import projects and documents, followed by the
associated links• Create references to class data
• Identify personnel responsible for the various parts of the data load process
Execution: Final Data Load• With proper planning and testing, the
final data load process “should” go smoothly
• Advices:
• Follow the documented procedures
• Allocate buffer time for unexpected issues
• Prepare to support users in the new system
Thank You!
Questions?
Marc YoungxLM Solutions, [email protected]
Script example shown is available at: http://www.xlmsolutions.com/COE2009/example.zip