Planning an Enterprise GeoDatabase Solutiondownloads.esri.com/.../Planning_an_Enterprise...Planning an Enterprise GeoDatabase Solution Planning an Enterprise GeoDatabase Solution Robert

Planning an Enterprise GeoDatabase Solution Planning an Enterprise Planning an Enterprise GeoDatabase Solution GeoDatabase Solution

Robert Kircher, Chris CushenberyEnterprise Implementation Services and Product Development

Robert Kircher, Chris CushenberyRobert Kircher, Chris CushenberyEnterprise Implementation Services and Product DevelopmentEnterprise Implementation Services and Product Development

AgendaAgenda

•• Establish Session Goals and ScopeEstablish Session Goals and Scope•• Review an Enterprise GeodatabaseReview an Enterprise Geodatabase•• Briefly Review Each Enterprise Geodatabase Briefly Review Each Enterprise Geodatabase

ComponentComponent•• Review Deployment ApproachesReview Deployment Approaches•• SummarizeSummarize•• QuestionsQuestions

Some GoalsSome Goals

•• Clarify key components and tasks needed to Clarify key components and tasks needed to build an enterprise Geodatabase.build an enterprise Geodatabase.

•• Clarify the definition and technologies behind Clarify the definition and technologies behind an enterprise Geodatabase.an enterprise Geodatabase.

•• Perhaps clear up some misconceptions about Perhaps clear up some misconceptions about the Geodatabasethe Geodatabase

•• Perhaps validate your existing plansPerhaps validate your existing plans•• Provide a high level Provide a high level CHECKLISTCHECKLIST

Scope and AssumptionsScope and Assumptions

•• A high level overview of components and tasks A high level overview of components and tasks •• Be practical, not academic (road tested)Be practical, not academic (road tested)•• Clear up some misconceptions about the Clear up some misconceptions about the

GeodatabaseGeodatabase•• Data design, not application designData design, not application design

Enterprise GeoDatabase Enterprise GeoDatabase Components and ServicesComponents and Services

…… the areas that you will consider in your planning.the areas that you will consider in your planning.

Enterprise GeoDatabase Enterprise GeoDatabase Components and InternalsComponents and Internals

ArcSDE/DBMSServer

G eoM etric N etworkT ab les

T opologyT ab les

B us iness T ab les

B us iness T ab les w ithS patial C olum n R as ter T ab les

ArcSDESystem Tables GeoDatabase

System Tables

ArcSDE API

GeometricNetworks

SubtypesTopology

Domains DisconnectedEditing

FeatureClasses

Rules

FeatureDatasets

T CP/IPSockets

Raster

Versioning

SpatialColumns

Logs

Locks

Geocoding

ApplicationStack

S earc hes

F etc h ing

M od ifying

Ins ertingS patia l

P roc es s ing

SQL

SQL

SQL

ArcObjects

ArcToolbox ArcCatalog ArcMap

Map and Business Products

Note: This is a general picture of theinternals. There may be someexceptions to this picture.

Why is an Enterprise Geodatabase Important?Why is an Enterprise Geodatabase Important?

•• Serves data promptly and efficientlyServes data promptly and efficiently•• Serves data to multiple users concurrently Serves data to multiple users concurrently •• Collaborative editing, and long transactions (versioning)Collaborative editing, and long transactions (versioning)•• Provides seamless dataProvides seamless data•• Secures data centrallySecures data centrally•• Provides an infrastructure for distributing and replicating Provides an infrastructure for distributing and replicating

datadata•• Build integrity into data maintenanceBuild integrity into data maintenance•• Provides a means to integrate spatial and Provides a means to integrate spatial and businesbusines datadata•• Provides a means to unite and leverage GIS and IT talent Provides a means to unite and leverage GIS and IT talent

and resources and resources •• Provides a means to cut business costs and improve Provides a means to cut business costs and improve

revenuerevenue•• …… and many othersand many others

Conceptual View of an Enterprise GIS SolutionConceptual View of an Enterprise GIS Solution

…… the enterprise GDB will touch almost every part of an enterpristhe enterprise GDB will touch almost every part of an enterprise e solution plansolution plan

DataMaintenance

Data Management

(GeoDatabase)

Work and Job Management

DataPublication

Plan Work

Schedule Work

Analysis Product

Data Product

Legacy Data Store

Legacy Product

Map Product

Report Work

Start Work

Complete Work

Monitor Work

Mobile Editors

Data Updates Convert

Data

Extract Data

QA/QC Editors

Capture Editors

Publication TierMaintenance Tier Product Tier

LoadData

Why plan an Enterprise GeoDatabase solution? Why plan an Enterprise GeoDatabase solution? (it(it’’s not so obvious)s not so obvious)

•• Enterprise Geodatabase projects are elaborate, and Enterprise Geodatabase projects are elaborate, and require diligent alignment with application effortsrequire diligent alignment with application efforts

•• Enterprise Geodatabases impact almost every part of Enterprise Geodatabases impact almost every part of building an enterprise solutionbuilding an enterprise solution

•• PMPM’’s need a means to cost and res need a means to cost and re--cost, budget, manage cost, budget, manage staff and deliverables, and mitigate riskstaff and deliverables, and mitigate risk

•• Enterprise Geodatabases cross boundaries, disciplines Enterprise Geodatabases cross boundaries, disciplines and staff (plans create a common goal) and staff (plans create a common goal) Building an Building an effective enterprise GeoDatabase is plain effective enterprise GeoDatabase is plain complicatedcomplicated–– Multiple stepsMultiple steps–– Multiple organizationsMultiple organizations–– Multiple risksMultiple risks

Some Common Enterprise Plans and ProjectsSome Common Enterprise Plans and Projects

•• Large SolutionsLarge Solutions–– Consultant team, dedicated project managementConsultant team, dedicated project management–– Custom applicationsCustom applications–– Large user baseLarge user base–– Elaborate, large databasesElaborate, large databases–– Multi phased approachMulti phased approach

•• Workgroup SolutionsWorkgroup Solutions–– Built in house, part time project managementBuilt in house, part time project management–– As much COTS functionality as possibleAs much COTS functionality as possible–– Evolve the Geodatabase, gradually move old to Evolve the Geodatabase, gradually move old to

newnew•• Mixed mode of old and new can work here (old Mixed mode of old and new can work here (old

ArcInfoArcInfo accessing a GDB)accessing a GDB)

•• All require planning All require planning ……

Approaches to Plans and ProjectsApproaches to Plans and Projects

•• Plan by datasetsPlan by datasets–– Start with a key data/dataset, and evolve the Start with a key data/dataset, and evolve the

functionality around it (great way to quickly get functionality around it (great way to quickly get traction, momentum, and experience) traction, momentum, and experience)

•• Plan by functionalityPlan by functionality–– Pick discrete parts of functionality, prioritize it (most Pick discrete parts of functionality, prioritize it (most

bang, simplest first), and build itbang, simplest first), and build it•• Plan by resources (budget, timeframe, and Plan by resources (budget, timeframe, and

staffing)staffing)–– Build what you can get Build what you can get ……

•• All require planning All require planning ……

Overview of the Major ComponentsOverview of the Major Components

•• Data ModelingData Modeling–– Conceptual and PhysicalConceptual and Physical–– What data and datasets? Parcels? Streets? What data and datasets? Parcels? Streets? ……–– What tables and data files?What tables and data files?–– Relationships? Topology? Relationships? Topology?

•• Data Procurement, Staging, and LoadingData Procurement, Staging, and Loading–– Where is my data now?Where is my data now?–– Convert it? Tools?Convert it? Tools?–– When and where to I stage it?When and where to I stage it?–– Is the data good?Is the data good?

•• Data MaintenanceData Maintenance–– How do I edit my data? How do I edit my data? –– Add more data?Add more data?–– Editing tools?Editing tools?

•• Data PerformanceData Performance–– Is my database serving data fast enough? Storing?Is my database serving data fast enough? Storing?–– How many users can my database support?How many users can my database support?–– What are reasonable performance expectations?What are reasonable performance expectations?

Major components Major components ---- contcont’’dd

•• Data AvailabilityData Availability–– Just business hours? 24x7? Epic Just business hours? 24x7? Epic ““Five 9Five 9’’ss””??–– Automated failAutomated fail--over options? Manual recovery?over options? Manual recovery?

•• Data RecoveryData Recovery–– Recovery options? DBMS backups and utilities? Recovery options? DBMS backups and utilities?

Various export and import tools?Various export and import tools?–– Recover responsiveness?Recover responsiveness?

•• Data AccessData Access–– Accessing the database? Intranet? Internet? Accessing the database? Intranet? Internet?

Mobile scenarios?Mobile scenarios?–– Means of access? Interfaces? ArcObjects? Means of access? Interfaces? ArcObjects?

ArcMap and ArcMap and ArcCatalogArcCatalog? SQL? ArcSDE C or Java ? SQL? ArcSDE C or Java APIs?APIs?

–– Interface maintenance? Versions compatibility?Interface maintenance? Versions compatibility?


•• Data monitoringData monitoring–– Who uses my data? How much?Who uses my data? How much?–– Most active data?Most active data?–– Database usage? Common queries? Operations? Database usage? Common queries? Operations?

Optimized accordingly? Optimized accordingly? –– Alerts? Notifications? System crashes? Slow Alerts? Notifications? System crashes? Slow

transactions? Security breaches?transactions? Security breaches?•• Data infrastructureData infrastructure

–– Hardware? Software? DBMS? Compatibility?Hardware? Software? DBMS? Compatibility?–– Centralize it? Distribute it?Centralize it? Distribute it?–– Licensing issues? (notice how I gave this an Licensing issues? (notice how I gave this an

exclusive bullet)exclusive bullet)–– Other services (recovery, availability, replication, Other services (recovery, availability, replication,

etc.)? etc.)? –– Development resources? Development resources? –– Configuration control?Configuration control?


•• Data replicationData replication–– Replicate? Why? Recovery? Performance? Replicate? Why? Recovery? Performance?

Mobility? Accessibility?Mobility? Accessibility?–– Architecture options?Architecture options?

•• Data distributionData distribution–– Sharing data? Internet (Sharing data? Internet (ArcIMSArcIMS)? Flat files )? Flat files

((shapefilesshapefiles)? Automated replication? Direct )? Automated replication? Direct connection to ArcSDE? Disconnected edits?connection to ArcSDE? Disconnected edits?

–– What data? How much?What data? How much?•• Data securityData security

–– Users?Users?–– Roles?Roles?

LetLet’’s s BrieflyBriefly Look at Each Component Look at Each Component ……

Data ModelingData Modeling


•• Building a structure for your data, including tables, relationshBuilding a structure for your data, including tables, relationships, ips, specification, metadata, identification, feature definition, specification, metadata, identification, feature definition, topology rules, etc.topology rules, etc.

•• Essential TasksEssential Tasks–– Gather requirementsGather requirements

•• Data ProductsData Products•• Map and Visualization ProductsMap and Visualization Products•• Analysis and Decision Support ProductsAnalysis and Decision Support Products•• Maintenance and Editing NeedsMaintenance and Editing Needs•• Metadata (editing and product)Metadata (editing and product)•• Spatial and Business Data IntegrationSpatial and Business Data Integration

–– Analysis and DesignAnalysis and Design•• Create conceptual data modelCreate conceptual data model

–– Identify data, metadata, specification, relationshipsIdentify data, metadata, specification, relationships•• Create physical data model, UMLCreate physical data model, UML

–– Identify GDB feature datasets, classes, relationships, domains, Identify GDB feature datasets, classes, relationships, domains, subtypes, geometric networks, etc.subtypes, geometric networks, etc.


•• Essential TasksEssential Tasks–– Conceptual ModelingConceptual Modeling

•• Document what will be in the spatial database, how the Document what will be in the spatial database, how the data will be maintained, interact, and be published data will be maintained, interact, and be published (conceptual modeling deliverables)(conceptual modeling deliverables)

–– Physical ModelingPhysical Modeling•• Document physical data model in a UML and codeDocument physical data model in a UML and code•• Build physical model into Geodatabase instance.Build physical model into Geodatabase instance.


•• Key DeliverablesKey Deliverables–– Requirements documentationRequirements documentation–– UML based data model or script/code generated data UML based data model or script/code generated data

modelmodel•• Challenges and RisksChallenges and Risks

–– Application development has Application development has critical dependencycritical dependency on on the modeling deliverablesthe modeling deliverables

–– Normalization balance (over versus deNormalization balance (over versus de--normalized)normalized)–– Changing the model downstream, propagating schema Changing the model downstream, propagating schema

changeschanges–– Thorough review of model among publication, Thorough review of model among publication,

maintenance, and vendor teamsmaintenance, and vendor teams–– Optimized for both publication and maintenance needsOptimized for both publication and maintenance needs

•• ESRI ResourcesESRI Resources–– ESRI library of essential, industry data modelsESRI library of essential, industry data models

Data Procurement and LoadingData Procurement and Loading

Data Procurement and LoadingData Procurement and Loading

•• Building a Building a ““one offone off”” loading system for the initial load of the loading system for the initial load of the GeodatabaseGeodatabase

•• Essential TasksEssential Tasks–– RequirementsRequirements

•• Identify data requirements (spatial, business)Identify data requirements (spatial, business)•• Identify conversion and translations requirementsIdentify conversion and translations requirements•• Identify data staging needsIdentify data staging needs•• Identify conversion automationIdentify conversion automation

–– Analysis and DesignAnalysis and Design•• Discover data sources (formats, internal, commercial)Discover data sources (formats, internal, commercial)•• Define conversion toolsetDefine conversion toolset

–– Simple, object data loadersSimple, object data loaders–– Model Builder, Model Builder, GeoprocessingGeoprocessing, Interoperability extension, Interoperability extension–– Custom data loaderCustom data loader–– ArcSDE data loaders (very simple data)ArcSDE data loaders (very simple data)

•• Create automationCreate automation•• Discover anticipated data volumes for storage and DBMS Discover anticipated data volumes for storage and DBMS

sizing (number, size of features)sizing (number, size of features)•• Define toolset and conversion methodsDefine toolset and conversion methods•• Discover special order to loadingDiscover special order to loading•• Create strategies for big dataCreate strategies for big data•• Define data verification/qc tools and methodsDefine data verification/qc tools and methods

Data Procurement and Loading contData Procurement and Loading cont’’dd

•• ImplementationImplementation–– Build data loading system for development and deploymentBuild data loading system for development and deployment

•• Procure dataProcure data•• Stage dataStage data•• Load dataLoad data•• View/use/verify dataView/use/verify data

•• Key DeliverablesKey Deliverables–– Requirements documentsRequirements documents–– Create toolsetCreate toolset–– Create loading automationCreate loading automation–– Pilot project (full cycle)Pilot project (full cycle)

•• Identify valid data subsetIdentify valid data subset•• procure dataprocure data•• stage itstage it•• load it load it •• Use and verify itUse and verify it

Data Procurement and Loading contData Procurement and Loading cont’’dd

•• Challenges and RisksChallenges and Risks–– Delivering to application development in a timely mannerDelivering to application development in a timely manner–– VersioningVersioning

•• PerformancePerformance•• reconciliation/postreconciliation/post•• compress compress

–– Scrubbing dataScrubbing data–– Managing changes to the loaders (full releases Managing changes to the loaders (full releases vsvs patch patch

approach)approach)–– Annotation generally requires special attention to loadAnnotation generally requires special attention to load–– Procuring entire dataset in a coordinated, timely manner Procuring entire dataset in a coordinated, timely manner –– Initial load duration could be significantInitial load duration could be significant

•• Capturing any changed data that occurred during load Capturing any changed data that occurred during load (depending on deployment approach)(depending on deployment approach)

Data MaintenanceData Maintenance

Data MaintenanceData Maintenance

•• Help configure and manage the maintenance Help configure and manage the maintenance workflow in the Geodatabase instance.workflow in the Geodatabase instance.

•• Essential Tasks Essential Tasks –– RequirementsRequirements

•• Identify maintenance workflow requirements (who, what, Identify maintenance workflow requirements (who, what, when, collaboration, automation, objects, batch when, collaboration, automation, objects, batch automation, etc.)automation, etc.)

•• Identify anticipated data volatilityIdentify anticipated data volatility–– Analysis and DesignAnalysis and Design

•• Define editing workflows (simple, cyclical, elaborate, Define editing workflows (simple, cyclical, elaborate, optimization opportunities beyond current workflow)optimization opportunities beyond current workflow)

•• Define reconcile, post, compress regimesDefine reconcile, post, compress regimes•• Define versioning structureDefine versioning structure•• Define specifics about edit volumes, version durations, Define specifics about edit volumes, version durations,

etc. that impact performanceetc. that impact performance

Data Maintenance contData Maintenance cont’’dd

–– ImplementationImplementation•• Configure versions in workflow or develop them Configure versions in workflow or develop them

into applicationinto application•• Build administration toolset (DBMS, GDB, Build administration toolset (DBMS, GDB,

compress, version folding, etc.)compress, version folding, etc.)

Data PerformanceData Performance

Data Performance and Data Performance and Scalability(QoSScalability(QoS))

•• Measure, assess, and optimize the performance of key Measure, assess, and optimize the performance of key functionality using the Geodatabase instance.functionality using the Geodatabase instance.

•• Essential TasksEssential Tasks–– Review anticipated data loadsReview anticipated data loads

•• Volume (data file growth management)Volume (data file growth management)•• Extent characteristics (spatial index tuning)Extent characteristics (spatial index tuning)•• Volatility (storage partitioning)Volatility (storage partitioning)

–– Identify key business transactionsIdentify key business transactions•• Maintenance operationsMaintenance operations•• Publication operationsPublication operations

–– Identify Identify QoSQoS requirements for key business transactionsrequirements for key business transactions•• response timeresponse time•• Initial and scheduled user loadsInitial and scheduled user loads•• throughputthroughput

Data Performance (Data Performance (QoSQoS))

•• Essential Tasks contEssential Tasks cont’’dd–– TestingTesting

•• Create test plan based on key operations and Create test plan based on key operations and requirementsrequirements

•• Develop testsDevelop tests•• Execute testsExecute tests•• Measure performance successMeasure performance success

–– Tune application, database server, and DBMSTune application, database server, and DBMS

Data Performance (Data Performance (QoSQoS) cont) cont’’dd

•• DeliverablesDeliverables–– Document requirementsDocument requirements–– Execute performance, analyze, optimize iterationsExecute performance, analyze, optimize iterations–– Tuning DBMS, tuning applicationTuning DBMS, tuning application–– Scaling strategyScaling strategy

•• Scale out Scale out vsvs upup•• Challenges and RisksChallenges and Risks

–– Sizing the spatial index optimallySizing the spatial index optimally–– Data too granular Data too granular

•• Group features Group features –– Overloading your applicationOverloading your application

•• Overloading application table of contentsOverloading application table of contents•• Building batchBuilding batch--like operations into applicationlike operations into application

–– ... many others (please attend the performance related GDB ... many others (please attend the performance related GDB sessions at the conference. This is an important topic).sessions at the conference. This is an important topic).

Data AccessData Access


•• Identify the right ArcSDE and DBMS interfaces (development or Identify the right ArcSDE and DBMS interfaces (development or off the shelf) for your application.off the shelf) for your application.

•• Essential TasksEssential Tasks–– Review DBMS and ArcSDE interfacesReview DBMS and ArcSDE interfaces

•• DBMS:DBMS: JDBC, SQL, OLEDB/ADO, ODBC, DBMS JDBC, SQL, OLEDB/ADO, ODBC, DBMS specific APIsspecific APIs

•• ArcSDE:ArcSDE: OLEDB/ArcObjects, C and Java API, SQLOLEDB/ArcObjects, C and Java API, SQL–– Identify database access and interface needsIdentify database access and interface needs

•• Mobile needsMobile needs•• Development environment (SQL, ArcObjects, Development environment (SQL, ArcObjects,

MapObjectsMapObjects, C or Java API) , C or Java API) •• Direct connect, traditional 3 tier, multi tier ArcSDE Direct connect, traditional 3 tier, multi tier ArcSDE configconfig


•• Essential TasksEssential Tasks–– Identify nonIdentify non--GIS application needsGIS application needs

•• GIS attribute dataGIS attribute data•• Business reports based on GIS data or Business reports based on GIS data or

processingprocessing–– Define and configure the application interfaces Define and configure the application interfaces

based on application needsbased on application needs•• Network configuration (host and ports)Network configuration (host and ports)•• Client libraries (e.g. Client libraries (e.g. SQLNetSQLNet, Java , Java libslibs, ArcSDE , ArcSDE

client client libslibs, etc.), etc.)

Data Access contData Access cont’’dd

•• DeliverablesDeliverables–– Document requirements and designDocument requirements and design–– Clients correctly configured to access ArcSDE Clients correctly configured to access ArcSDE

instanceinstance•• Challenges and RisksChallenges and Risks

–– Compatibility issuesCompatibility issues–– Interface limitations (no interface can do all Interface limitations (no interface can do all

business and GIS application operations)business and GIS application operations)

Data MonitoringData Monitoring


•• Monitoring the usage of your Geodatabase, and notifying of any Monitoring the usage of your Geodatabase, and notifying of any anomalous behavioranomalous behavior

•• Essential TasksEssential Tasks–– Identify alerting and notification needsIdentify alerting and notification needs

•• Functional related alertsFunctional related alerts–– DBMS crashDBMS crash–– Query activityQuery activity

•• Performance related alertsPerformance related alerts–– System usageSystem usage–– System loadsSystem loads–– SQL query loadsSQL query loads

•• Notification tools and infrastructureNotification tools and infrastructure–– EmailEmail–– TelephonyTelephony


•• Essential Tasks contEssential Tasks cont’’dd–– Configure alerts and notificationsConfigure alerts and notifications–– Automate alert responses where practicalAutomate alert responses where practical–– Measure data volumes and volatilityMeasure data volumes and volatility

•• DeliverablesDeliverables–– Document of alerts and notification needsDocument of alerts and notification needs–– Procedures, tools, scripts for creating alertsProcedures, tools, scripts for creating alerts–– Capture and alerting infrastructureCapture and alerting infrastructure

Data Monitoring contData Monitoring cont’’dd

•• Challenges and RisksChallenges and Risks–– Configuring and maintaining an elaborate Configuring and maintaining an elaborate

infrastructure of alertsinfrastructure of alerts–– Overly alerted (i.e. a system that cries wolf). Overly alerted (i.e. a system that cries wolf). –– Creating a responsive system to alerts (automated Creating a responsive system to alerts (automated

and manual) and manual)

Data SecurityData Security


•• Making the Geodatabase instance secure from Making the Geodatabase instance secure from unauthorized usersunauthorized users

•• Essential TasksEssential Tasks–– Review DBMS authentication schemesReview DBMS authentication schemes

•• Integrated with OS and network domain Integrated with OS and network domain securitysecurity

•• Standard DBMS securityStandard DBMS security•• Mixed modeMixed mode•• Users and rolesUsers and roles

–– Identify anticipated users (GIS and business Identify anticipated users (GIS and business applications), and accessible objects (spreadsheet applications), and accessible objects (spreadsheet sitting here)sitting here)


•• Essential Tasks contEssential Tasks cont’’dd–– Define and create DBMS usersDefine and create DBMS users–– Define and create DBMS rolesDefine and create DBMS roles–– Create security monitorCreate security monitor–– Update DBMS security patchesUpdate DBMS security patches

•• DeliverablesDeliverables–– Document of users, roles, and object access.Document of users, roles, and object access.–– Configured users and rolesConfigured users and roles

Data Security contData Security cont’’dd

•• Challenges and RisksChallenges and Risks–– Sharing a DBMS login Sharing a DBMS login

•• SDE_logfileSDE_logfile contention pointcontention point•• NoteNote

–– ItIt’’s easier to grant access to users later, than it is s easier to grant access to users later, than it is to revoke later.to revoke later.

Data InfrastructureData Infrastructure


•• Building the hardware and software infrastructure for the Building the hardware and software infrastructure for the Geodatabase instance, and all the related data servicesGeodatabase instance, and all the related data services


•• Identify hardware and software requirements based functional Identify hardware and software requirements based functional and system needsand system needs

–– Development and testDevelopment and test–– ProductionProduction–– LicensingLicensing–– System capacity and growthSystem capacity and growth–– Storage needsStorage needs–– Host CPU, RAMHost CPU, RAM–– Network bandwidthNetwork bandwidth


•• Essential Tasks contEssential Tasks cont’’dd–– Requirements Requirements

•• Identify configuration control requirements and Identify configuration control requirements and plans over development database and serverplans over development database and server

–– Software development databasesSoftware development databases–– Data scrubbing databasesData scrubbing databases–– PrePre--production/integration databaseproduction/integration database

–– ImplementationImplementation•• Procure and configure hardware and softwareProcure and configure hardware and software

–– development and testdevelopment and test–– productionproduction

Data Infrastructure contData Infrastructure cont’’dd

–– ImplementationImplementation•• Create ArcSDE and DBMS instancesCreate ArcSDE and DBMS instances

–– Development, Test, and Deployment Development, Test, and Deployment

Data Infrastructure contData Infrastructure cont’’dd

•• DeliverablesDeliverables–– Configured development and test environmentConfigured development and test environment–– Configured production environmentConfigured production environment–– Create and enforce configuration control planCreate and enforce configuration control plan

•• Challenges and RisksChallenges and Risks–– Creating nimble development and test Creating nimble development and test

environments (quickly change)environments (quickly change)–– Controlled and predictable configuration controlControlled and predictable configuration control–– Difficult to size and plan for capacity early on Difficult to size and plan for capacity early on

(there are mitigation tactics for this, scale out/up)(there are mitigation tactics for this, scale out/up)

Data RecoveryData Recovery


•• Building a reliable and responsive recovery system Building a reliable and responsive recovery system into the Geodatabase instance into the Geodatabase instance

•• Essential TasksEssential Tasks–– Review and understand recovery optionsReview and understand recovery options

•• DBMS backup resourcesDBMS backup resources•• DBMS import/exportDBMS import/export•• 33rdrd party DBMS solutionsparty DBMS solutions•• ArcToolboxArcToolbox conversion tools (various flat files)conversion tools (various flat files)•• ArcSDE ArcSDE sdeimportsdeimport/sdeexport/sdeexport•• GDB copy/pasteGDB copy/paste•• Disconnected editingDisconnected editing•• Incremental Incremental vsvs full backupsfull backups


•• Essential Tasks contEssential Tasks cont’’dd–– RequirementsRequirements

•• Identify recovery requirementsIdentify recovery requirements–– Acceptable time to full/partial recoveryAcceptable time to full/partial recovery–– Database to recoverDatabase to recover–– Application configuration/data recoveryApplication configuration/data recovery

–– Analysis and DesignAnalysis and Design•• Define and create recovery proceduresDefine and create recovery procedures•• Define and create recovery tools, scripts and Define and create recovery tools, scripts and

automation automation –– ImplementationImplementation

•• Create recovery toolset, and automationCreate recovery toolset, and automation

Data Recovery contData Recovery cont’’dd

•• DeliverablesDeliverables–– Document requirements and designDocument requirements and design–– Document recovery proceduresDocument recovery procedures

•• Challenges and RisksChallenges and Risks–– Not ensuring backup actually works, testing it Not ensuring backup actually works, testing it

regularly.regularly.–– Time constraints will not allow full backup, only Time constraints will not allow full backup, only

deltasdeltas–– Stringing together backup changesStringing together backup changes–– Disk size limitationsDisk size limitations–– Automation fraught with complicationsAutomation fraught with complications–– Recovering application specific data, configuration Recovering application specific data, configuration

filesfiles

Data ReplicationData Replication


•• Replicating data from the Geodatabase instance to Replicating data from the Geodatabase instance to subscribing users or systems.subscribing users or systems.

•• Essential TasksEssential Tasks–– Review and understand common replication Review and understand common replication

configurationsconfigurations•• SnapshotSnapshot•• MultiMulti--master/mergemaster/merge•• TransactionalTransactional•• HybridHybrid

–– Review replication optionsReview replication options•• Device level, OS level, DBMS levelDevice level, OS level, DBMS level•• 33rdrd party solutionsparty solutions•• ArcGIS solutions (disconnected editing, ArcGIS solutions (disconnected editing,

extracts, distributed replication, archiving)extracts, distributed replication, archiving)



•• Identify replication uses and benefitsIdentify replication uses and benefits–– Performance/load balancingPerformance/load balancing–– MobilityMobility–– RecoveryRecovery–– AvailabilityAvailability–– Network load reduction (be careful here)Network load reduction (be careful here)

•• Identify data to be replicatedIdentify data to be replicated•• Identify Identify QoSQoS requirementsrequirements

–– how fast should changes replicate?how fast should changes replicate?–– how frequent is acceptable?how frequent is acceptable?

Data Replication contData Replication cont’’dd

•• Essential Tasks contEssential Tasks cont’’dd–– Analysis and DesignAnalysis and Design

•• Define replication architectureDefine replication architecture–– ImplementationImplementation

•• Prototype and test architecture (crucial)Prototype and test architecture (crucial)–– Key data modificationsKey data modifications–– Typical and peak loadsTypical and peak loads

•• Procure, install, and configure replication Procure, install, and configure replication architecturearchitecture

–– TestTest•• Test replication accuracy and responsivenessTest replication accuracy and responsiveness


•• DeliverablesDeliverables–– Document requirements and designDocument requirements and design–– Full cycle of prototypingFull cycle of prototyping

•• Procure and configure replication Procure and configure replication software/hardwaresoftware/hardware

•• Build master databaseBuild master database•• Modify data, and measure success and Modify data, and measure success and

performance of replica performance of replica –– Configured and tested replication systemConfigured and tested replication system


•• Challenges and RisksChallenges and Risks–– Realizing benefits, Realizing benefits, ““saving reads, not writessaving reads, not writes””–– BandwidthBandwidth–– Volume of data modificationsVolume of data modifications–– Multi master or merge replication is extremely Multi master or merge replication is extremely

difficult to manage and usedifficult to manage and use–– Snapshot is usefulSnapshot is useful–– Choosing BLOB storage over LONG RAW Choosing BLOB storage over LONG RAW

(Oracle)(Oracle)

Data Availability (Data Availability (QoSQoS))


•• Keeping the Geodatabase instance available for use Keeping the Geodatabase instance available for use to your enterprise.to your enterprise.

•••• Essential TasksEssential Tasks

–– Review and understand availability optionsReview and understand availability options•• Server clusteringServer clustering•• DBMS clustering (share data files, but not DBMS clustering (share data files, but not

processes)processes)•• 3rd party solutions3rd party solutions•• Replication architectureReplication architecture•• manual backup and restoremanual backup and restore

–– RequirementsRequirements•• Identify uptime requirementsIdentify uptime requirements

–– 8hrsx5days, 24x7, 99.999% of year?8hrsx5days, 24x7, 99.999% of year?–– Mean time between failureMean time between failure


•• Essential TasksEssential Tasks–– Analysis and DesignAnalysis and Design

•• Define availability architectureDefine availability architecture–– ImplementationImplementation

•• Configure availability architectureConfigure availability architecture–– TestTest

•• Test availability failure scenarios against Test availability failure scenarios against QoSQoSrequirementsrequirements

Data Availability (Data Availability (QoSQoS) cont) cont’’dd

•• DeliverablesDeliverables–– Document requirements and designDocument requirements and design–– Design, procure, and configure availability Design, procure, and configure availability

architecturearchitecture–– TestingTesting

•• Challenges and RisksChallenges and Risks–– Identifying and mitigatingIdentifying and mitigating

•• singlesingle--point of failures in your data systempoint of failures in your data system•• developing redundant systems developing redundant systems

–– clustering may not be enough clustering may not be enough –– Not consistently testing systemNot consistently testing system

Data DistributionData Distribution


•• Build an infrastructure to distribute data from the Build an infrastructure to distribute data from the Geodatabase instance to users or systems.Geodatabase instance to users or systems.

•• Essential TasksEssential Tasks–– Review and understand distribution optionsReview and understand distribution options

–– GDB to GDB, GDB to GDB to GDB, GDB to shapefileshapefile, GDB to , GDB to coverage coverage

–– Disconnected editing, distributed replicationDisconnected editing, distributed replication–– DBMS export fileDBMS export file–– ArcIMSArcIMS, ArcGIS Server, ArcGIS Server–– Replication services (see replication)Replication services (see replication)

–– RequirementsRequirements•• Identify data consumersIdentify data consumers

–– ArcGIS applicationsArcGIS applications–– Custom Custom ArcObjectArcObject appapp



•• Identify delivery needsIdentify delivery needs–– Response time and throughputResponse time and throughput–– Data volumeData volume–– Distribution medium (network, DVD, etc.)Distribution medium (network, DVD, etc.)

•• Identify distribution infrastructure requirementsIdentify distribution infrastructure requirements–– BandwidthBandwidth–– WAN/LAN/wirelessWAN/LAN/wireless–– CapacityCapacity–– Security issuesSecurity issues

Data Distribution contData Distribution cont’’dd

•• DeliverablesDeliverables–– Document of data distribution needsDocument of data distribution needs

•• ConsumersConsumers•• DeliveryDelivery•• DestinationDestination

–– Configured distribution mechanismsConfigured distribution mechanisms•• Challenges and RisksChallenges and Risks

–– Bandwidth limitations for moving dataBandwidth limitations for moving data–– Size of dataSize of data–– Advanced data types (annotation, Advanced data types (annotation, geonetworksgeonetworks))

Other Planning ObservationsOther Planning Observations

Deployment ApproachesDeployment Approaches

•• ““Big BangBig Bang””–– Release the entire system at once (data, Release the entire system at once (data,

functionality, staff, etc.)functionality, staff, etc.)•• Geographic RolloutGeographic Rollout

–– Users by geographic regionsUsers by geographic regions–– Data by geographic regionsData by geographic regions

•• Functional RolloutFunctional Rollout–– Release by discrete functionalityRelease by discrete functionality

•• Dataset RolloutDataset Rollout–– Release by datasetRelease by dataset

Other Planning IssuesOther Planning Issues

•• TrainingTraining–– Introduction to editing and versioningIntroduction to editing and versioning–– Introduction enterprise Geodatabase Introduction enterprise Geodatabase

administrationadministration•• ArcSDE admin toolsArcSDE admin tools•• DBMS adminDBMS admin

–– Introduction to enterprise Geodatabase Introduction to enterprise Geodatabase application developmentapplication development•• ArcObjectsArcObjects•• MapObjectsMapObjects•• ArcSDE C and Java APIArcSDE C and Java API•• SQLSQL

Other Planning IssuesOther Planning Issues

•• Organizational Roles and Responsibilities Organizational Roles and Responsibilities –– The Enterprise GeoDatabase technically unites The Enterprise GeoDatabase technically unites

the organization, but requires new, more the organization, but requires new, more collaborative relationships to resolve issues like:collaborative relationships to resolve issues like:

•• RecoveryRecovery•• SecuritySecurity•• Anomaly resolutionAnomaly resolution•• Performance tuningPerformance tuning•• DevelopmentDevelopment

•• Put your database team within reach of your Put your database team within reach of your development staffdevelopment staff

Profiles of Poor Plans Profiles of Poor Plans ……

–– Do NOT allow enough time and care given to data Do NOT allow enough time and care given to data procurement and loadingprocurement and loading

–– Do NOT carefully profile user loads and performance Do NOT carefully profile user loads and performance expectations expectations

–– Over commit replication to move dataOver commit replication to move data•• packed word and conceptpacked word and concept•• Usually difficult to configure and manageUsually difficult to configure and manage•• Usually brittleUsually brittle

–– Overloading ArcSDE/GDB with bad application decisionsOverloading ArcSDE/GDB with bad application decisions•• Too many requestsToo many requests•• Too much data volumeToo much data volume

–– Do NOT align correctly with data consumers during Do NOT align correctly with data consumers during development and deploymentdevelopment and deployment

–– Poor deployment analysis and planPoor deployment analysis and plan

Key MilestonesKey Milestones

–– Initial data model deliveredInitial data model delivered–– Initial data procurement and staging Initial data procurement and staging

delivereddelivered–– Initial data loaders deliveredInitial data loaders delivered–– Initial data load deliveredInitial data load delivered–– Initial results from performance testing key Initial results from performance testing key

operations and loads deliveredoperations and loads delivered–– First complete pass through the lifecycle of First complete pass through the lifecycle of

a typical data edit delivered (start editor to a typical data edit delivered (start editor to customer receipt)customer receipt)

Quick Demo: Draft of Generalized Quick Demo: Draft of Generalized Project PlanProject Plan

Additional ResourcesAdditional Resources

•• For more detailed discussion of the components: For more detailed discussion of the components: –– Other sessionsOther sessions–– White papers White papers

•• ESRI WebsiteESRI Website•• ESRI Support WebsiteESRI Support Website

–– Product documentationProduct documentation–– ESRI staff at the conferenceESRI staff at the conference–– Enterprise Implementation ServicesEnterprise Implementation Services–– ESRI Enterprise Advantage Program (EEAP)ESRI Enterprise Advantage Program (EEAP)

Open to QuestionsOpen to QuestionsContact info:Contact info:

Robert KircherRobert [email protected]@esri.com

Chris Chris [email protected]@esri.com

Documents

Planning an Enterprise GeoDatabase Solutiondownloads.esri.com/.../Planning_an_Enterprise...Planning an Enterprise GeoDatabase Solution Planning an Enterprise GeoDatabase Solution Robert