Scribe Software CDK Developer Guide - Amazon Web Services · The API takes advantage of the latest Microsoft technologies, including .NET 4.0, Silverlight, Enterprise Service Bus,

1

Contents Getting Started .......................................................................................................... 3

About Scribe Online and Connectors .......................................................................... 3 Scribe Online Services .............................................................................................. 3 CDK Components .................................................................................................... 3 Audience ................................................................................................................ 4 Prerequisites ........................................................................................................... 4 Requirements ......................................................................................................... 4 CDK Workflow ......................................................................................................... 5

Scribe Online Architecture ........................................................................................... 6 Application Architecture............................................................................................ 6 Connector Roles ...................................................................................................... 6

Key Concepts ............................................................................................................. 8 Scribe Online Replication Services (RS) ................................................................... 8 Scribe Online Synchronization Services (SYS) .......................................................... 8 RS Source............................................................................................................ 8 RS Target ............................................................................................................ 8 SYS Source / SYS Target ....................................................................................... 8 Administrator ....................................................................................................... 9 User .................................................................................................................... 9

Design Considerations .............................................................................................. 10 SYS Connectors ..................................................................................................... 10 RS Source Connectors ............................................................................................ 10 RS Target Connectors ............................................................................................ 11

Sample Projects ....................................................................................................... 12 Using the RS Source Sample Connector .................................................................... 13 Using the RS Target Sample Connector .................................................................... 13 Using the SYS Sample Connector ............................................................................ 14 Setting Up the Sample Database ............................................................................. 14 About the Sample Project Unit Tests ........................................................................ 15

Testing ................................................................................................................... 16 Unit Testing .......................................................................................................... 16

Overview of the Unit Testing Process ..................................................................... 16 Example Code .................................................................................................... 16 Interpreting Test Results ..................................................................................... 17

Attached Debugger Testing ..................................................................................... 17 Setting up Verbose Logging .................................................................................... 18

User Acceptance Testing ........................................................................................... 19 Developing Connectors ............................................................................................. 20

Enabling the Scribe Online API in your Development Environment ............................... 20 Enabling the Scribe Online API in Microsoft Visual Studio ............................................ 21

Connector Basics ...................................................................................................... 22 Developing Replication Service Sources .................................................................... 22

RS Source Connector Overview ............................................................................ 23 RS Source Methods ............................................................................................. 24 UI Generated ..................................................................................................... 25 Connection ........................................................................................................ 26 Role .................................................................................................................. 26 Testing and Debugging ........................................................................................ 26 Setting up Verbose Logging ................................................................................. 27 RS Source Pseudo Code ....................................................................................... 28

Developing Replication Services Targets ................................................................... 30

2

RS Target Connector Overview ............................................................................. 31 RS Target Methods ............................................................................................. 31 UI Generated ..................................................................................................... 33 Connection ........................................................................................................ 33 Role .................................................................................................................. 33 Testing and Debugging ........................................................................................ 34 Setting up Verbose Logging ................................................................................. 34 RS Target Pseudo Code ....................................................................................... 35

Developing a Synchronization Service ...................................................................... 37 SYS Connector Overview ..................................................................................... 38 SYS Methods ...................................................................................................... 38 Connection UI Generation .................................................................................... 40 Connection ........................................................................................................ 40 Role .................................................................................................................. 41 Testing and Debugging ........................................................................................ 41 Setting up Verbose Logging ................................................................................. 41 SYS Pseudo Code................................................................................................ 42

3

Getting Started

Welcome to the Scribe Online Connector Development Kit (CDK). This CDK includes fully

functional sample connector projects, unit test programs, and reference documentation for

the Scribe Online API.

About Scribe Online and Connectors

Scribe Online is a cloud-based integration platform that enables customers and partners to

integrate applications in cloud-to-cloud environments, or in hybrid environments—

integrating a cloud-based system with an on-site application.

Scribe Online is architected to provide a core integration engine and an extensible set of connectors. Connectors enable Scribe Online to communicate with specific applications using the appropriate technology and API for each application.

Connectors insulate the Scribe Online integration engine from the specific technologies needed to communicate with each application. Connectors also enable the integration engine to provide a common interface experience to end-users, allowing them to focus on the business processes required to complete an integration rather than the specific technologies involved.

Scribe Online Services

The Scribe Online integration platform provides multiple services. Connectors need to use different methods in the Scribe Online API depending upon the services you choose.

Scribe Online Replication Service (RS) provides basic data replication and gives you the ability to easily maintain a copy of your source data (in a proprietary format such as Microsoft Dynamics CRM or Salesforce.com) in a Microsoft SQL Server database.

Scribe Online Synchronization Service (SYS) provides two-way integration between source and target data stores. Scribe Online SYS enables you to map, filter, and transform data from Microsoft Dynamics CRM, Salesforce.com or Microsoft SQL Server as part of your data integration.

CDK Components

This CDK includes the following major components:

Getting Started, Scribe Online Architecture, and Design Considerations

Developing Connectors and Connector Basics

Complete the sample connector projects in ScribeOnlineCDKSamples.zip

Unit Testing

Connector Reference

https://openmind.scribesoftware.com/download/ScribeOnlineCDKSamples

4

Audience

This document has two intended audiences:

1. Business Analysts or Architects who need to understand the capabilities of the Scribe Online CDK.

2. Developers who need to implement a Scribe Online Connector for their application.

Prerequisites

Scribe recommends that the connector developers be competent in .NET development and have a thorough understanding of the application API or technology being exposed to Scribe Online. Developers will also benefit from ancillary skills with web services, relational databases, and testing.

Additional skills should include general API / SDK development using C++, C#, or Visual Basic; and supporting tools. All sample code and examples are written in C#.

Developers do not need to have any previous experience with Scribe Online technologies, architecture, or internal data structures.

Requirements

The development computer must meet the following requirements:

Processor (Pentium or equivalent): 800 MHz (Minimum); 1GHz (Recommended)

Memory: 1 GB (Minimum); 4 GB (Recommended)

Hard Disk: Up to 100 GB of available space

Display: 800 x 600, 256 colors (Minimum); 1024 x 768 high color, 32-bit (Recommended)

Operating Systems: Windows 7, Windows Server 2003, Windows Server 2008, Windows Server 2008 R2.

Web Browsers: Microsoft Internet Explorer 7.x, 8.x,9.x; Mozilla Firefox 4.x, 5.x, 6.x; Google Chrome 15.

To develop a Scribe Online Connector and make use of the CDK, you must have the following development environment available:

Microsoft Visual Studio 2010

Microsoft .Net 4.0

The sample project scripts require SQL server. If you do not have access to an SQL Server

instance, download a free copy of Microsoft SQL Server Express Edition.

http://www.microsoft.com/sqlserver/en/us/editions/express.aspx

5

CDK Workflow

To develop connectors using this CDK, complete the following major steps in order:

1. Identify how your connector will be used to determine what Scribe Online role to assign to it. For more information see Connector Roles. All connectors need to be assigned a role. A role determines which Scribe Online services the connector can work with and whether the connector can be used as a source, as a target, or both.

2. Implement the methods required to support the connector roles. There are specific methods that need to be used in a connector depending on the connector’s role.

3. Exercise unit tests to make sure the new connectors are functionally accurate and complete. This CDK includes unit testing programs to help you test your connector before you deploy it to use with Scribe Online and before you do user acceptance testing.

4. Deploy your connector locally so you can test it with Scribe Online. Deploying requires you to copy the connector to a specific folder on the computer where your Scribe Online agent is installed. Then after restarting your Scribe Online agent, Scribe Online will recognize your connector and let you use it in Scribe Online.

5. Complete User Acceptance Testing to make sure the connectors meet your expectations and the user’s needs in the target environment. Scribe recommends that you test the connector by using it in several Scribe Online solutions that exercise all of the connector capabilities. This is the best way to know that your connector is working properly with Scribe Online and that it meets all of your requirements.

6. Distribute the connector. When your connector is ready for general use, you can make it available to your customers.

6

Scribe Online Architecture

Scribe Online is a cloud-based integration platform that enables customers and partners to build integrations between any combination of cloud-based and on-premise applications.

The API takes advantage of the latest Microsoft technologies, including .NET 4.0, Silverlight, Enterprise Service Bus, Entity Framework, RIA Services and Windows Identify Framework. Scribe Online is hosted using Microsoft Azure Services.

Scribe Online is architected to provide a core integration engine and an extensible set of connectors. Connectors enable Scribe Online to communicate with specific applications using the most appropriate technology and API for each application.

Connectors insulate the Scribe Online integration engines from the details of any specific technology involved. Connectors also enable the integration engines to expose a common user experience to end-users, helping them focus on building the business processes required for a particular integration rather than the details of specific technologies involved.

Application Architecture

The following graphic shows the interaction between the components of a typical Scribe Online deployment. The Scribe Online API provides a means for developers to create custom connectors joining the components of a Scribe Online deployment.

Connector Roles

Scribe Online uses a concept called Connector Roles to identify different ways of using connectors. A connector’s role is defined by the developer of the connector and is based on how they want the connector to be used. When Scribe Online loads a connector, the role indicates where the connector is selectable for use in the Scribe Online user interface. The role also defines which methods need to be implemented from the Scribe Online API. Connectors are not limited to a single role; in-fact a Synchronization Service connector is required to have

7

two roles (Scribe.IS.Target and Scribe.IS.Source). If a connector is available for both Replication Service (Scribe.RS.Target and Scribe.RS.Source) and Synchronization Service roles it is up to the connector developer to decide if the data store and their architecture of the connector can handle multiple role capabilities.

8

Key Concepts

You should understand the following Scribe Online concepts before you begin developing your own connector.

Scribe Online

Scribe Online provides data integration capability to the user in the form of solutions. Each solution is designed enable a user to solve a set data integration problem, such as data replication or data synchronization. Users can configure any number instances from any of the available solutions within the context of a solution instance.

Scribe Online Replication Services (RS)

The Scribe Online Replication Service solution (RS) enables you to replicate data

from either Microsoft Dynamics CRM 2011 Online, other cloud-based applications

such as Salesforce, or on-premise deployments; to a Microsoft SQL Server database.

Scribe Online Synchronization Services (SYS)

The Scribe Online Synchronization Service solution (SYS) allows you to synchronize data between various online and on-premise data stores. SYS offers entity mapping, field transformation, source filtering and net change, and support for the standard create, read, update, and delete operations.

Connector Roles

You must develop connectors to support one or more of the following roles:

RS Source

The Scribe Online RS Source is the origin of the data to be replicated and can be any data store that you want to replicate. Connectors with this role are available in Source connection selection for creation or editing of Scribe Online RS solutions.

RS Target

The Scribe Online RS Target is the destination of the data to be replicated and can be any data store that you want to store the replicated data. Connectors with this role are available in Target connection selection for Scribe Online RS solutions.

SYS Source / SYS Target

The Scribe Online IS Source and IS Target may be used as both a destination and origin of data to be synchronized using synchronization services (SYS). Connectors with these two roles are available for selection as a source and as a target in Scribe Online SYS solutions

Agents

Scribe Online Agents facilitate communication between the data stores used by a solution. The Agent provides secure communications to Scribe Online without opening security holes in your corporate firewall. The Agent communicates with your source and target systems as well as with Scribe Online in the cloud.

9

Connections

A connection is the login and other parameters you need to connect to any data store used by Scribe Online. Every connection is named, to allow you to organize and track your Scribe Online connections. Each connection provides access to a single data store; such as a Microsoft Dynamics CRM Organization or a SQL Server database.

Organizations

An Organization represents the company (or other entity) that is using this Scribe Online Solution.

Users

Scribe Online has two different user roles:

Administrator

A Scribe Online Administrator has full access to Scribe Online capability, including tasks such as creating and managing Organizations and Solutions Instances, installing Agents, and inviting other users to join Organizations.

User

Non-administrative users become members of an Organization by being invited to join by an Administrator. Users can perform many of the tasks that Administrators can, including creating and managing Organizations and Solution Instances, and installing Agents. However, a Scribe Online User cannot make changes to the Organization or its members.

10

Design Considerations

SYS Connectors

Scribe Online SYS Connectors complete the four fundamental create-read-update-delete (CRUD) operations. An SYS Connector parses metadata to determine if an object allows any or all of the CRUD operations.

SYS Connectors require the following conditions:

The metadata must provide an object’s child-to-parent relationship information so that complex queries can be generated and executed through the connector.

The metadata must provide property-level available actions, such as whether a field is read-only, has a default value, and must be included in a create operation.

Update and delete operations must allow multiple objects to be accessed and modified with one call to the connector.

Query executions to the connector must allow queried data to be organized in either ascending or descending order, for any specified field. For example, a DateTime field must be able to provide records newest to oldest in ascending order and oldest to newest in descending order.

Query executions from the connector must be able to query against a root object for any and all data from the root object, and/or one or all related parent object data.

RS Source Connectors

Determine whether or not the data source you are connecting to already has a process in place to track changes in data. Specifically, does the data source use a timestamp to indicate a change in data? If the data source does not have a process for tracking changes, can a process be implemented via connector interaction with the data source? Connectors require a single timestamp per record that is changed the last time the data was synchronized to retrieve all records created, updated, or deleted since the last replication.

If a process is not already in place to track data changes using a timestamp, you can implement change tracking, on either a global level and/or an object-by-object basis. For example: if the source is a database and does not track any changes, the connector must make supporting schema changes.

Before replicating, execute InitReplication to create a Change History table. The Change History table tracks table names, timestamps, and primary key values. This table is updated by triggers created using InitReplicationObject on each table in the database. InitReplicationObject can also create table columns to track the date that the record was last modified. If no changes are required, returning success in either the InitReplication or InitReplicationObject method is sufficient.

When a connector retrieves metadata, it must specify which field in the source contains the time stamp. The GetReplicationData method retrieves all replication data since the last time the

11

data was synchronized, and passes it using the method input properties from Scribe Online. If this is an initial replication, it sends the minimum date of 1/1/0000. The connector must determine whether or not to change this date, if it is not supported. In the example provided, the SQL database supports a minimum date of 1/1/1753 so the minimum date is reset to use this value.

When a connector retrieves data from a source, it must be converted into a common data type to enable transfer between systems. Connectors must use .NET standard data types as a universal means to transfer data.

RS Source connectors must convert data to .Net standard data types.

RS Target connectors receive data as .NET data types, and must convert data to the expected format of the target system receiving the replicated data.

RS Target Connectors

Before each replication, the agent compares metadata between the source and target connections. The target connector is responsible for detecting and handling changes in metadata.

For example: if the target is a database, the target connector executes the CreateOrUpdateObjectForReplication method to detect changes such as a new column in the table. The table must be deleted and recreated with the new schema information. During replication, the table is refilled with the new data. If a new table was added, this method would also be responsible for creating a table that does not exist.

12

Sample Projects

To help you develop connectors, Scribe Software has created a sample database and

several sample projects available in a zip file: ScribeOnlineCDKSamples.zip.

Extract the contents of the ScribeOnlineCDKSamples.zip file to your hard drive and browse

for the following folders:

\Binaries\ — Contains the Scribe.Core.ConnectorApi file as well as all dependent libraries. Please reference the Scribe.Core.ConnectorApi to access the API features and include the supporting files with your project.

\SampleRSSourceDatabase\ — Contains SQL scripts that you can run from the Query Browser in Microsoft SQL Server Management Studio to create and populate a sample source database. You can use this database to test the ability of your RS Source connectors to communicate with a commonly used source.

\Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.RS_Source\ — Contains a complete sample connector for an RS source. You can use this connector as a template to develop RS source connectors.

\Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.Source.RS_Test\ — Contains C# unit tests for each Replication Service source connector class in the API: metadata providers, methods, operations, and connections. The test scripts validate both positive and negative use cases. You can use these scripts to confirm that your connectors are working properly.

\Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.RS_Target\ — Contains a complete sample connector for an RS target. You can use this connector as a template to develop RS target connectors.

\Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.RS_Target.Test\ — Contains C# unit tests for each Replication Service target connector class in the API: methods, operations, and connections. The test scripts validate both positive and negative use cases. Use these scripts to confirm that your connectors are working properly.

\Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.SYS\ — Contains a complete sample connector for Synchronization Service. You can use this connector as a template to develop SYS connectors.

\Scribe.Connector.Cdk.Sample\Scribe.Connector.Cdk.Sample.SYS.Test\ — Contains C#

unit tests for each Synchronization Service connector class in the API: metadata providers, operations, queries and connections. The test scripts validate both positive and negative use cases. Use these scripts to confirm that your connectors are working properly.


13

Using the RS Source Sample Connector

The RS Source sample connector can be used with Scribe Online for exploration through user acceptance testing scenarios or through unit testing scenarios. You must first compile the RS Sample source project.

If you are using the sample connector for exploration by using it with Scribe Online, you must deploy the connector by copying the .dll file to a directory under where the Scribe Online Agent is installed.

If you are using the sample connector for exploration through unit testing scenarios, you do not have to copy the .dll file because it is referenced by the unit test project.

Regardless of how you are using the RS Source sample connector, you must copy the following files into the directory where the RS Source sample connector resides:

ScribeChangeHistory_Create.sql

ScribeDelete_Trigger.sql

These files are located in the \Samples\SampleRSSourceDatabase\ directory.

You must also set up the sample database as described in Setting up the Sample Database.

Using the RS Target Sample Connector

The RS Source target connector can be used with Scribe Online for exploration through user acceptance testing scenarios or through unit testing scenarios. You must first compile the RS Target project.

If you are using the sample connector for exploration by using it with Scribe Online, you must deploy the connector by copying the .dll file to a directory under where the Scribe Online Agent is installed.

If you are using the sample connector for exploration through unit testing scenarios, you do not have to copy the .dll file because it is referenced by the unit test project. The unit testing project references the sample database. If the sample database is not installed, you must install it to unit test the RS Sample Target connector. Refer to “Setting up the sample database”.

If you are going to use the RS Target Sample Connector in Scribe Online in a user acceptance testing scenario or in an attached debugger scenario, you must create an empty SQL Server database and you must have login credentials for a user that has full rights to the database.

14

Using the SYS Sample Connector

The CDK includes a sample connector for SYS that functions as both a source and a target. You can use the sample SYS connector with Scribe Online for exploration through user acceptance testing scenarios or through unit testing scenarios. To use the SYS connector you must first compile the SYS Connector project.

If you are using the sample connector for exploration by using it with Scribe Online, you must deploy the connector by copying the .dll file to a directory under where the Scribe Online Agent is installed. In this scenario you can use the connector to connect to any SQL Server database.

If you are using the sample connector for exploration through unit testing scenarios, you do not have to copy the .dll file because it is referenced by the unit test project. The unit testing project references the sample database. If the sample database is not installed, you must install it to unit test the RS Sample Target connector. See Setting up the Sample Database for details.

Setting Up the Sample Database

Before setting up the database, make sure the supporting database applications are installed as described in the Requirements.

To create the sample database needed to test Scribe Online connectors:

1. Start the Query Browser in Microsoft SQL Server Management Studio and open the \Samples\SampleRSSourceDatabase\ folder.

2. Run the first script SampleRSSource.sql. This script creates the sample database, ScribeSampleRSSource.

3. Run the second script SampleRSSourceData.sql. This script populates the database with sample data. You can repeat this step any time during testing, such as after testing the delete connector, to remove all previous changes and repopulate the database.

Note: You must run these scripts in order. The order of operations mirrors how the Scribe Online production databases are created and populated. The database change history is required because new records are automatically discovered, but finding deleted records without a change history would require an inefficient and time-consuming line-by-line comparison between the source and target databases.

15

About the Sample Project Unit Tests

Sample test code (when executed) only makes changes to the ScribeSampleRSSource database. The test code does not alter any other files.

Some tests can only be run once (per database refresh) in order to return useful results. For example, a delete test deletes a database record, which cannot be deleted again. To refresh the sample database between tests, repeat Step 3 in Setting Up the Database.

Test connectors make changes to both source and target databases. The source tests create a trigger for a specified table. This trigger updates the change history table with information about each deleted row in the specified database. The test code also deletes rows from the change history table once the deleted information has been sent to the target.

The target requires an empty SQL database in a location that can be accessed and discovered by the sample test code. This database must be referenced in the connection information parameter through the connect method. The target has the ability to create and update table schema, as well as perform all create-read-update-delete (CRUD) operations.

16

Testing

Scribe recommends that connector developers follow a testing process to verify that the connector is giving the expected results. The best practice for this is to do three levels of testing: unit testing, attached debugger testing, and user acceptance testing. This CDK provides a number of resources as examples to help you set up this testing process.

Unit Testing

This CDK provides sample projects that you can use as a model to set up unit tests. You can use the unit test projects included with the CDK that work with the sample connector projects and adapt the unit test projects to your connector. The sample unit test projects include tests for metadata, methods, operations, and connection. See Sample Projects for more information.

Overview of the Unit Testing Process

Complete the following process to develop and run tests:

Download and open the Sample Projects.

Create unit tests for methods Setting Up Tests.

Test on the code Running Tests.

View test results and fix errors in the code Interpreting Test Results.

Note: You can build and run the sample connectors and unit tests included in the CDK. Refer to the Scribe Sample Projects and the Microsoft Developer Network Testing site for guidance.

Example Code

Scribe provides extensive unit tests for each of the CDK sample projects. These tests include examples of all expected method inputs and return values for the Scribe.Core.ConnectorApi, IConnector, and IMetadataProvider interfaces.

This example code shows the basics elements of a connector’s Connect() method for testing purposes. View any of the sample project code for more details.

/// <summary> ///This is to test the third-party connection /// </summary> [TestMethod] public void SourceConnectorConnectionValidTest() { //create a new instance of the sample connector var sourceConnector = new SourceConnector(); //setup the initial parameters for data connection Dictionary<string, string> properties = new Dictionary<string, string> { {"Provider", "SQLNCLI10"}, {"Server", "localhost"}, {"Database", "ScribeSampleRSSource"},

http://msdn.microsoft.com/en-us/library/aa292484%28v=VS.71%29.aspx

17

{"UserName", "sa"} }; //encrypt the connection password using the shared key string encryptedPassword = Encryptor.Encrypt_AesManaged("sa", CryptoKey); properties.Add("Password", encryptedPassword); //call the connect method from the connector and pass //in the connection properties dictionary sourceConnector.Connect(properties); //do a check that the IsConnected flag is true //and the connection has been opened Assert.IsTrue(sourceConnector.IsConnected); }

Interpreting Test Results

When the test is complete, see the Test Results window for information about the success

or failure of the test. For more information, see Analyzing Test Results in the MSDN Library.

Attached Debugger Testing

In addition to Unit Tests, connectors may be tested by attaching the connector project to the running Scribe Online agent process. This allows for precise testing of a connector that is being used by a running Scribe Online solution instance.

To perform attached debugger testing, download and install the Scribe Online Agent from Scribe Online.

To deploy your connector:

1. Stop the Scribe Online Agent Windows service. 2. Navigate to one of the following folders, depending upon your operating system:

For 32-bit (X86) ̶ \Program Files\Scribe Software\Scribe Online Agent\Connectors\ -OR- For 64-bit (X64) ̶ \Program Files (x86)\Scribe Software\Scribe Online Agent\Connectors\

3. In this folder, create a new subfolder for the connector to test, such as Scribe.Connector.Cdk.Sample.Source or any unique name.

4. Copy all files built in debug mode from the Visual Studio project into the newly created folder.

For example, Scribe.Connector.Cdk.Sample.RS_Source includes the following files in its folder:

Scribe.Connector.Cdk.Sample.RS_Source.dll

Scribe.Connector.Cdk.Sample.RS_Source.pdb

ScribeChangeHistory_Create.sql

ScribeDelete_Trigger.sql

http://msdn.microsoft.com/en-us/library/ms182493(v=vs.90).aspx

http://scribesoft.com/Online

18

5. Start the Scribe Online Agent Windows service. 6. Create a connection using the newly installed connector. It should appear in the list of

available connections in the Scribe Online user interface. 7. Set up a Scribe Online solution that uses the new connection. 8. With the connector solution open in Visual Studio:

a. Select Debug and Attach to Process. b. From the list of running processes, select Scribe.Core.ProcessorService.exe. c. Click Attach.

See the Scribe Online Help Center for more information about working with Scribe Online

Agents, which describes installing an agent, creating connections, and creating a solution

instance.

Setting up Verbose Logging

All Scribe Online connectors include extensive logging messages that provide an accurate and detailed description of what occurs during replication. By default, installing a Scribe Online Agent activates standard logging.

To turn one verbose logging:


For 32-bit (X86) ̶ \Program Files\Scribe Software\Scribe Online Agent\

-OR- For 64-bit (X64) ̶ \Program Files (x86)\Scribe Software\Scribe Online Agent\

3. Open the file Scribe.Core.ProcessorService.exe.config in NotePad with Administrator privileges, or make sure you are logged in as an account with Administrator privileges before opening the file.

4. Set Debug=true in the Scribe.Logger.LogEntrySeverities application setting. The edited line should match the following example:

<add key="ScribeLogger.LogEntrySeverities" value="None=true,Debug=true,Info=true,Warning=true,Fatal=true,Error=true" />

5. Save and close the file. 6. Start the Scribe Online Agent Windows service.

http://www.scribesoft.com/helpcenter/prod/online/en/

19

User Acceptance Testing

User acceptance testing should be completed after unit testing, and attached debugger testing. Perform user acceptance testing on a different computer from the development computer.

Install the connector as described in the Testing and Debugging section for each connector.

Make sure the connector shows up in Scribe Online and only in the places appropriate to its declared role. For example, if the connector is for RS Source only, then it should only show up in an RS solution on the source connection list.

Use the connector in a Scribe Online solution and make sure it performs as expected:

o Connection dialog works and you can connect.

o Connection data persists and can be edited.

o Provides metadata.

o Metadata is refreshed when schema changes are made against that connection.

o If the connector is a source connector for SYS, does it provide source data? Does the net change feature work? Does the filter feature work?

o If the connector is a source connector for SYS and DB has relationships defined, make sure that the parent relationships can be used in mapping.

o If the connector is a target connector for SYS, does it perform all the CRUD operations it should support?

o If the connector is a target for SYS, does it shows correct actions/operations the entity can support (part of metadata)?

o If the connector is a source for RS, does it replicates read appropriate data into a target connection on both INITIAL replication and subsequent replications, with inserts, updates, and deletes in the source prior to the next run?

o If the connector is a source for RS, can the user can select custom entities to be replicated?

o If the connector is a target for RS, make sure that both initial and subsequent replications are replicated in Target RS?

20

Developing Connectors

To develop your own Scribe Online Connectors, use Microsoft .NET tools and technologies. Each Connector consists of two basic components, the Connector itself, which is installed and runs on the Scribe Online Agent, and a User Interface (UI) component which can collect connection information from the user.

A minimal connector requires that you implement two interfaces, requiring a total of 16 combined properties and methods. For this release of Scribe Online, Connector developers will have access to a dynamically generated user interface, which creates a UI similar to the following. Each connector uses the PreConnect method to generate this interface.

Enabling the Scribe Online API in your Development Environment

The Scribe Online API comes in the form of .dll files that you must reference from your development environment. The API includes the following files:

Scribe.Core.Common.dll – Provides basic Scribe Online functionality. Place a copy of

this file in the build directory for your connectors.

Scribe.Core.ConnectorAPI.dll – Provides connector-specific functionality.

Note: These files must exist in the same directory.

21

Enabling the Scribe Online API in Microsoft Visual Studio

1. Create a new development project from Visual Studio. 2. In the Solution Explorer tree, right-click on References and select Add Reference. The Add

Reference dialog box opens. 3. Select the Browse tab and navigate to the folder containing the API .dll files. 4. Select Scribe.Core.ConnectorApi.dll and click OK. The Add Reference dialog box closes and

the .dll file now appears listed in the references for the project. Visual Studio now has access to the components of the Scribe Online API.

22

Connector Basics

When developing a Connector for Scribe Online RS, you must provide functionality to support using the Connector as either a replication source or a target. This section describes the Connector Interfaces and both the RS Source Methods , RS Target Methods , and SYS Methods.

For a complete listing of the classes, interfaces, and enumerated lists that make up the Scribe Online Connector API see Scribe Online Connector Reference. The API, template connectors, and test scripts are available at:


Connecter Interfaces

The Scribe Online CDK uses two interfaces:

IMetadataProvider — Provides the metadata details to the Scribe Online RS Source Connector.

IConnector — Provides the runtime link between Scribe Online and the targeted application API or technology.

While both interfaces must be fully implemented, many of the methods can be “stubbed out” depending on if you are developing an RS Source or Target Connector. The sections below detail what methods must be implemented for each type of Connector.

Developing Replication Service Sources

The Scribe Online RS Source is the origin of the data to be replicated and can be any data store that you want to replicate. Before developing an RS source connector, determine whether or not the data source you are connecting to already has a process in place to track data changes. Specifically, does the data source use a timestamp to indicate a change in data? If the data source does not have a process for tracking changes, can a process be implemented via connector interaction with the data source? Connectors require a single timestamp per record that is changed the last time the data was synchronized to retrieve all records created, updated, or deleted since the last replication.

If a process is not already in place to track data changes using a timestamp, you can implement change tracking, on either a global level and/or an object-by-object basis. For example: if the source is a database and does not track any changes, the connector must make supporting schema changes.

Before replicating, execute InitReplication to create a Change History table. The Change History table tracks table names, timestamps, and primary key values. This table is updated by triggers created using InitReplicationObject on each table in the database. InitReplicationObject can also create table columns to track the date that the record was last modified. If no changes are required, returning success in either the InitReplication or InitReplicationObject method is sufficient.


23

When a connector retrieves metadata, it must specify which field in the source contains the time stamp. The GetReplicationData method retrieves all replication data since the last time the data was synchronized, and passes it using the method input properties from Scribe Online. If this is an initial replication, it sends the minimum date of 1/1/0000. The connector must determine whether or not to change this date, if it is not supported. For example, if the SQL database supports a minimum date of 1/1/1753, the minimum date is reset to use this value.

When a connector retrieves data from a source, it must be converted into a common data type to enable transfer between systems. Connectors must use .NET standard data types as a universal means to transfer data.

RS Source connectors must convert data to .NET standard data types.

RS Target connectors receive data as .NET data types, and must convert data to the expected format of the target system receiving the replicated data.

RS Source Connector Overview

The following graphic details the interaction between the various methods, the agent, and a source connector:

24

RS Source Methods

To develop a Scribe Online RS Source Connector, implement the RS Source methods for each Interface.

An RS Source Connecter requires the following interfaces and properties:

IConnector.PreConnect

IConnector.Connect

IConnector.Disconnect

IConnector.ExecuteOperation

IConnector.ExecuteMethod

IMetadataProvider.RetrieveObjectDefinitions

IConnector.Preconnect

Preconnect is called before a connection has been established so that information can be exchanged between the Connector and its UI. Currently, this method is used solely to allow the adapter to configure the dynamic Connector UI that will be used for all Connectors.

IConnector.Connect

Connect instructs the Connector to connect to its data store. The Connector can choose to maintain or drop this connection as desired while the Connector is in use. At a minimum, implement this method to test the Connection when called.


Disconnect is called to instruct the Connector to disconnect from its data store and free any resources that may be associated with the connection.


The ExecuteOperation method performs one or more operations, such as Create, Update or Delete, on a given data object type (such as an Account). The OperationInput object that is passed in provides the operation name, object name, data values and other information needed to perform the operation. The returned OperationResult object returns the results of the operation, including data or error information as needed.

To support the Scribe Online RS Source role, only the Delete operation is required.

Delete Operation — ScribeOnline executes this method after synchronization has

occurred. If data stored in the change history object is no longer needed it will be

removed using the Delete operation. The ‘LastSyncDate’ determines the range of

records that need to be removed and is provided by Scribe Online through the

LookupCondition (part of OperationInput object.)

If this feature is not available in this Connector, create a placeholder method that

returns success.

25


ExecuteMethod is called by Scribe Online to perform any Connector-specific logic. Unlike operations, which always operate on a single object type such as an Account, the ExecuteMethod methods are capable of operating on and returning any number of different object types. For information about implementing this method, see the Scribe Online Connector Development Guide.

For a Connector to support the Scribe Online RS Source role, this method must support the following ExecuteMethod source methods:

GetObjectDefinition — Get the specified 'Object' name, from the data source. Include

the primary key or identifying item in the each of the objects.

GetObjectDefinitionList — Get the list of 'Object' names, (table names, in this case)

from the data source. Include the primary key or identifying item in the each of the

objects.

GetReplicationData — Scribe Online executes this source side method to retrieve data

that will be replicated. Three pieces of information are provided to this method from

Scribe Online that use properties found in the MethodInput object:

ObjectName — For SQL, the table that is requested for replication.

ModificationDateFullName — The field name containing the time that the record

was last updated.

LastSyncDate — The date the system was last synchronized.

GetChangeHistoryData — This method is executed by ScribeOnline to retrieve change

history information from a specific object. The MethodInput provides the name of the

object that the information is being retrieved from (ObjectName) as well as the last

synchronization date (LastSyncDate).

InitReplication — This method is executed by ScribeOnline to create an Object that

keeps track of changes in the data source. For SQL, this object will be filled using

created triggers. If triggers are not available for this connector, have the method return

a success in the method result every time.

InitReplicationObject — This method is executed by Scribe Online to create a trigger

function inside the data source that will be used to keep track of changes. This method

should also be able to detect if the trigger already exists as well as call the

InitReplication method to ensure that the object for replication exists. If this is not

possible for this connector, the method should return a success in the method result

every time.


RetrieveObjectDefinitions is a required method called by Scribe Online to ask the Connector for a complete list of metadata object definitions. For example, a SQL Connector must provide table information that includes all columns and properties for each column.

UI Generated

Scribe Online Connector API allows for generation of connector specific UI. The call for the

generic UI is done through the IConnector.PreConnect method. The developer must use the

FormDefinition provided in the ConnectionUI namespace. Please see the sample connectors

for an example of usage.

26

Connection

When Scribe Online requests a connection the information required in the connection UI is sent through the method IConnector.Connect. It is up to the connector to handle the information that it requires for connecting to the origin of the data including decryption of sensitive data and establish a connection. Once the connection has been established, the IConnector.IsConnected flag must be set. The connector must also handle any retry logic that may be required if a connection fails or times out.

Role

The role of the Connector indicates what methods are being implemented from Scribe.Core.ConnectorApi, whether the connector will be accessing the data’s source, the data’s origin and which scribe solution type it will support. Roles are specified through the supportedSolutionRoles parameter of ScribeConnectorAttribute. For replication service source connectors add Scribe.RS.Source to the array of supported solution roles. When the Connector is recognized by Scribe Online, the supported solution role will be recognized and ConnectorTypeName specified for the connector will only show up in the appropriate connection user interfaces. For example, if the role of a connector is Scribe.RS.Source, then the ConnectorTypeName will only appear for replication services and may only be used as a source connector. Scribe Online does allow for multiple roles in a connector. SYS connectors, for example, must support both Scribe.IS.Source and Scribe.IS.Target roles, and may support one or both of the RS roles.

Note: Scribe Online does allow for multiple roles in a connector, in fact SYS connectors must support both Scribe.IS.Source and Scribe.IS.Target roles, and may support one or both of the RS roles.

Testing and Debugging

Once you have compiled your connector, you might want to install it and then test it. What follows here is an overview of how to do execute those processes. Refer to the Testing section to learn more about testing using the sample connector projects.

To install a connector:



3. In this folder, create a new subfolder for the connector to test. 4. Copy all files built in debug mode from the Visual Studio project into the newly created

folder. 5. Start the Scribe Online Agent Windows service. 6. The connector appears in the list of available connections in the Scribe Online user

interface.

27

7. Set up a connection with the newly installed connector. 8. Set up a Scribe Online solution that uses the new connection.

See the Scribe Online Help Center for more information about working with Scribe Online Agents, which describes installing an agent, creating connections, and creating a solution instance.


All Scribe Online connectors include extensive logging messages, including basic and verbose logging, which provide an accurate and detailed description of what occurs during replication. You can see how this type logging is implemented by exploring the sample connector projects. When connectors are running, they inherit the logging settings of the agent. By default, installing a Scribe Online Agent activates standard logging.

To turn on verbose logging for the agent and therefore your connector:


For 32-bit (X86) ̶ \Program Files\Scribe Software\Scribe Online Agent\ -OR- For 64-bit (X64) ̶ \Program Files (x86)\Scribe Software\Scribe Online Agent\

3. Open the file Scribe.Core.ProcessorService.exe.config in Notepad with Administrator privileges, or make sure you are logged in as an account with Administrator privileges before opening the file.





28

RS Source Pseudo Code

To help you understand the flow of the Scribe Online RS solution, refer to the following pseudo code. This example illustrates the flow of the RS process and the interaction between the RS Source and Target Connectors:

void MainProcess() { // connect SourceConnector.Connect(); TargetConnector.Connect(); // inform source that we are starting replication SourceConnector.InitReplication(null); // get a list of the object types to be replicated var objectDefList = SourceConnector.GetObjectDefinitionList(null); foreach(var objectDef in objectDefList) { // make sure target replication object has been created and is current var sourceFullObjectDef = SourceConnector.GetObjectDefinition(objectDef); var targetFullObjectDef = TargetConnector.GetObjectDefinition(objectDef); if (targetFullObjectDef == null) // target replication object not found { // inform source that we are replicating this object SourceConnector.InitReplication(objectDef); // create target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); } else if (targetFullObjectDef != sourceFullObjectDef) { // update target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); } // call next process to replicate the data for the current object ReplicateData(objectDef); } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); } void ReplicateData(object objectDef) { // get the timestamp of the last entity synced from the target var lastReplicationDate = TargetConnector.GetLastReplicationSyncDate(objectDef); // query the source for all data changed since the lastReplicationDate

29

var dataEntities = SourceConnector.GetReplicationData(lastReplicationDate); // walk the query results replicating the data foreach(var dataEntity in dataEntities) { // attempt to insert the data into the target var success = TargetConnector.Create(dataEntity); if(!success) // insert failed, attempt to replace the row { // remove TargetConnector.Delete(dataEntity); // replace success = TargetConnector.Create(dataEntity); // log an error on failure if (!success) { LogRowError(objectDef); } } } // call next process to soft delete any records deleted since last execution ProcessDeletes(objectDef, lastReplicationDate); }

void ProcessDeletes(object objectDef,DateTime lastReplicationDate) { if (lastReplicationDate != DateTime.MinValue) // we have valid replication data { // query the source for all deleted data since the lastReplicationDate var deletedEntities = SourceConnector.GetChangeHistoryData(objectDef, lastReplicationDate); // walk the query results marking the deleted records as deleted in the target foreach (var deletedEntity in deletedEntities) { TargetConnector.Update(deletedEntity); } } else // either an initial sync or an object with no last modified timestamp field { // update all records not updated by the current execution as // deleted in the target TargetConnector.Update(objectDef, lastReplicationDate); } }

30

Developing Replication Services Targets

RS target connectors are responsible for providing entity level, and property level metadata information to Scribe Online after initial synchronization. For each entity the connector must also specify a property that gets updated when a change occurs to a record. If this property does not exist in the destination then one must be created. This date is important to identify the point at which the last replication was run for the given entity. Before each replication, the agent compares metadata between the source and target connections. The target connector is responsible for detecting and handling changes in metadata. On initial replication it is up to the target connector to create a copy of the source metadata provided by Scribe Online.

For example: if the target is a database, the target connector executes the CreateOrUpdateObjectForReplication method to detect changes such as a new column in the table. The table must be deleted and recreated with the new schema information. During replication, the table is refilled with the new data. If a new table was added, this method would also be responsible for creating a table that does not exist.

31

RS Target Connector Overview

The following graphic details the interaction between the various methods, the agent, and a target connector:

RS Target Methods

To develop a Scribe Online RS Target Connector, implement each of the following IConnector methods:


IConnector.Connect






32

IConnector.Connect





ExecuteOperation is called by Scribe Online to perform one or more operations, such as create, update, or delete, on a given data object type (an Account, for example). For the Scribe Online RS Target role, you must implement all three operations (create, update, and delete). These operations are passed to the OperationInput object which stores the filtering information for that operation. These LookupConditions may contain information from a simple ‘where’ clause to multiple comparison operations.

The IConnector.ExecuteOperation target operations are:

Create — The Create operation allows a new row of data to be inserted into the target data source. Create is called when new data has been detected in the source during replication.

Delete — The Delete operation is executed by Scribe Online when an entity (row) of data has been deleted from the source data. Using properties stored in the Operation input this operation must allow for multiple deletions from a single Object (Table).

Update — The Update operation is executed when a data entity (row) has been changed in the RS source data. This operation uses filtering defined by the OperationInput.LookupCondition properties to update the selected fields.


Scribe Online calls ExecuteMethod to perform Connector-specific logic. Unlike operations which always operate on a single object type such as an Account, methods are capable of consuming and returning any number of different object types. The MethodInput object that is passed in provides the method name and other information needed to execute the method. The returned MethodResult object returns the results of the method execution, including data or error information as needed.

The IConnector.ExecuteMethod target methods are:

GetObjectDefinition — ScribeOnline executes this method to retrieve a specific Object's definition, which includes any attributes and supporting object properties.

CreateOrUpdateObjectForReplication — ScribeOnline executes this method to create or update an object that is being replicated. The method input will provide the definition of the object and the property definitions in the MethodInput parameter.

33

Three things are considered when developing this method. First check if the object already exists and create a new one if it does not. Second compare the property definitions against the existing object and update the object when a change has been detected. Lastly if no change is detected no changes to the data source is made and note in the MethodInput that this is the case by defining a property SchemaChanged and setting it to false.

GetLastReplicationSyncDate — Scribe Online executes this target side method to retrieve the last date data was replicated for a specific object. Two pieces of information are provided to this method from ScribeOnline use properties found in the MethodInput object. First the ObjectName in the case of SQL this would be the table that is requested for replication. Second is the ModificationDateFullName this is the column or field name that an update time for a row of data has been updated.

UI Generated

Scribe Online Connector API allows for generation of connector specific UI. The call for the generic UI is done through the IConnector.PreConnect method. The developer must use the FormDefinition provided in the ConnectionUI namespace. See the sample connectors for an example of usage.

Connection

When Scribe Online requests a connection the information required in the connection UI is sent through the method IConnector.Connect. It is up to the connector to handle the information that it requires for connecting to the origin of the data including decryption of sensitive data and establish a connection. Once the connection has been establish the IConnector.IsConnected flag must be set. The connector must also handle any retry logic that may be required if a connection fails or times out.

Role

The role of the Connector indicates what methods are being implemented from Scribe.Core.ConnectorApi, whether the connector will be accessing the data’s source, the data’s origin and which scribe solution type it will support. Roles are specified through the supportedSolutionRoles parameter of ScribeConnectorAttribute. For replication service target connectors add Scribe.RS.Target to the array of supported solution roles. When the Connector is recognized by Scribe Online the supported solution role will be recognized and ConnectorTypeName specified for the connector will only show up in the appropriate connection user interfaces. For example, if the role of a connector is Scribe.RS.Source then the ConnectorTypeName will only appear for replication services and may only be used as a source connector. Scribe Online does allow for multiple roles in a connector, in fact SYS connectors must support both Scribe.IS.Source and Scribe.IS.Target roles, and may support one or both of the RS roles.

34


Once you have compiled your connector, Scribe recommends installing and testing the new connector. This section provides an overview of how to complete this process. Refer to the Testing section to learn more about testing using the sample connector projects.






interface. 7. Set up a connection with the newly installed connector. 8. Set up a Scribe Online solution that uses the new connection.



All Scribe Online connectors include extensive logging messages, including basic and verbose logging, which provides an accurate and detailed description of what occurs during replication. You can see how this type logging is implemented by exploring the sample connector projects. When connectors are running, they inherit the logging settings of the agent. By default, installing a Scribe Online Agent activates standard logging.







35



RS Target Pseudo Code

To help you understand the flow of the Scribe Online RS solution, refer to the following pseudo code. This example illustrates the flow of the RS process and the interaction between the RS Source and Target Connectors:

void MainProcess() { // connect SourceConnector.Connect(); TargetConnector.Connect(); // inform source that we are starting replication SourceConnector.InitReplication(null); // get a list of the object types to be replicated var objectDefList = SourceConnector.GetObjectDefinitionList(null); foreach(var objectDef in objectDefList) { // make sure target replication object has been created and is current var sourceFullObjectDef = SourceConnector.GetObjectDefinition(objectDef); var targetFullObjectDef = TargetConnector.GetObjectDefinition(objectDef); if (targetFullObjectDef == null) // target replication object not found { // inform source that we are replicating this object SourceConnector.InitReplication(objectDef); // create target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); } else if (targetFullObjectDef != sourceFullObjectDef) { // update target replication object TargetConnector.CreateOrUpdateObjectForReplication(sourceFullObjectDef); } // call next process to replicate the data for the current object ReplicateData(objectDef); } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); }

36

void ReplicateData(object objectDef) { // get the timestamp of the last entity synced from the target var lastReplicationDate = TargetConnector.GetLastReplicationSyncDate(objectDef); // query the source for all data changed since the lastReplicationDate var dataEntities = SourceConnector.GetReplicationData(lastReplicationDate); // walk the query results replicating the data foreach(var dataEntity in dataEntities) { // attempt to insert the data into the target var success = TargetConnector.Create(dataEntity); if(!success) // insert failed, attempt to replace the row { // remove TargetConnector.Delete(dataEntity); // replace success = TargetConnector.Create(dataEntity); // log an error on failure if (!success) { LogRowError(objectDef); } } } // call next process to soft delete any records deleted since last execution ProcessDeletes(objectDef, lastReplicationDate); }

void ProcessDeletes(object objectDef,DateTime lastReplicationDate) { if (lastReplicationDate != DateTime.MinValue) // we have valid replication data { // query the source for all deleted data since the lastReplicationDate var deletedEntities = SourceConnector.GetChangeHistoryData(objectDef, lastReplicationDate); // walk the query results marking the deleted records as deleted in the target foreach (var deletedEntity in deletedEntities) { TargetConnector.Update(deletedEntity); } }

37

else // either an initial sync or an object with no last modified timestamp field { // update all records not updated by the current execution as // deleted in the target TargetConnector.Update(objectDef, lastReplicationDate); }

}

Developing a Synchronization Service

Scribe Online SYS Connectors complete the four fundamental create-read-update-delete (CRUD) operations. An SYS Connector parses metadata to determine if an object allows any or all of the CRUD operations.

SYS Connectors require the following conditions:

The metadata must provide an object’s child-to-parent relationship information so that complex queries can be generated and executed through the connector.

The metadata must provide property-level available actions, such as whether a field is read-only, has a default value, and must be included in a create operation.

Update and delete operations must allow multiple objects to be accessed and modified with one call to the connector.

Query executions to the connector must allow queried data to be organized in either ascending or descending order, for any specified field. For example, a DateTime field must be able to provide records newest to oldest in ascending order and oldest to newest in descending order.

Query executions from the connector must be able to query against a root object for any and all data from the root object, and/or one or all related parent object data.

38

SYS Connector Overview

The following graphic details the interaction between the various methods, the agent, and a synchronization connector:

SYS Methods

To develop a Scribe Online SYS Synchronization Connector, implement each of the following IConnector methods:


IConnector.Connect



IConnector.ExecuteQuery

IConnector.GetMetadataProvider

IMetadataProvider.RetrieveActionDefinitions


IMetadataProvider.RetrieveObjectDefinition

IMetadataProvider.ResetMetadata

39



IConnector.Connect





ExecuteOperation is called by Scribe Online to perform one or more operations, such as Create, Update or Delete, on a given data object type (an Account, for example). For the Scribe Online SYS role, you must implement all three operations (create, update, and delete). These operations are passed to the OperationInput object which stores the filtering information for that operation. These LookupConditions may contain information from a simple where clause to multiple comparison operations.

The IConnector.ExecuteOperation target operations are:

Create — The Create operation allows a new row of data to be inserted into the target data source. Create is called when new data has been detected in the source during replication.

Delete — The Delete operation is executed by Scribe Online when an entity (row) of data has been deleted from the source data. Using properties stored in the Operation input this operation must allow for multiple deletions from a single Object (Table).

Update — The Update operation is executed when a data entity (row) has been changed in the source data. This operation uses filtering defined by the OperationInput.LookupCondition properties to update the selected fields.

IConnector.ExecuteQuery

ExecuteQuery is called by Scribe Online as a request for specific data from the connector. The query object received by this method will include constraint information as well as a root entity.

Queries executed through this must allow for:

Filtering on a one or more property in the root entity.

A sequence in ascending or descending order on a one or more property in the root entity.

Include any information from one or more parent relationships of the root entity.

40

IConnector.GetMetadataProvider

GetMetadataProvider is a required method called by Scribe Online to retrieve an active instance of the Object that inherits the IMetadataProvider interface. This allows Scribe Online to access the IMetadataProvider specific methods.

IMetadataProvider.RetrieveActionDefinitions

RetrieveActionDefinitions is a required method called by Scribe Online to ask the connector which KnownActionTypes are supported in the connector on a global level. Some examples of know action types are Create, Update, Delete. If the connector does not support Updates, then none of the ObjectDefinitions returned from the connector may allow for the Update action. Also if the connector does not support this action then IConnector.ExecuteOpertion is not required to support the Update operation.


RetrieveObjectDefinitions is a required method called by Scribe Online to ask the Connector for a complete list of metadata object definitions. For example, a SQL Connector must provide table information that includes all columns and properties for each column, all child to parent foreign key relationships, and if the table allows for one/many or all crud operations.

IMetadataProvider.RetrieveObjectDefinition

RetrieveObjectDefinition is a required method called by Scribe Online to ask the Connector for a single metadata object definition. For example, a SQL Connector must provide table information that includes all columns and properties for each column, all child to parent foreign key relationships, and if the table allows for one/many or all crud operations.

IMetadataProvider.ResetMetadata

ResetMetadata is a required method called by Scribe Online to inform the connector that the

user has made a request to refresh metadata on demand. If metadata is being cached it

must be deleted and re-populated from the third-party connection. If metadata is being

received on demand a simple return is acceptable. Please see the sample connectors for the

non-cached implementation.

Connection UI Generation

Scribe Online Connector API allows for generation of connector specific UI. The call for the

generic UI is done through the IConnector.PreConnect method. The developer must use the

FormDefinition provided in the ConnectionUI namespace. Please see the sample connectors

for an example of usage.

Connection

When Scribe Online requests a connection the information required in the connection UI is sent through the method IConnector.Connect. It is up to the connector to handle the information that it requires for connecting to the origin of the data including decryption of sensitive data and establish a connection. Once the connection has been established, the IConnector.IsConnected flag must be set. The connector must also handle any retry logic that may be required if a connection fails or times out.

41

Role

The role of the Connector indicates what methods are being implemented from Scribe.Core.ConnectorApi, whether the connector will be accessing the data’s source, the data’s origin and which scribe solution type it will support. Roles are specified through the supportedSolutionRoles parameter of ScribeConnectorAttribute. For synchronization service connectors add Scribe.IS.Source AND Scribe.IS.Target to the array of supported solution roles. When the Connector is recognized by Scribe Online the supported solution role will be recognized and ConnectorTypeName specified for the connector will only show up in the appropriate connection user interfaces.


Once you have compiled your connector, Scribe recommends installing and testing the new connector. This section provides an overview of how to complete this process. Refer to the Testing section to learn more about testing using the sample connector projects.






interface. 7. Set up a connection with the newly installed connector. 8. Set up a Scribe Online solution that uses the new connection.



All Scribe Online connectors include extensive logging messages, including basic and verbose logging, that provide an accurate and detailed description of what occurs during replication. You can see how this type logging is implemented by exploring the sample connector projects. When connectors are running, they inherit the logging settings of the agent. By default, installing a Scribe Online Agent activates standard logging.


1. Stop the Scribe Online Agent Windows service.


42

2. Navigate to one of the following folders, depending upon your operating system:






SYS Pseudo Code

To help you understand the flow of the Scribe Online SYS solution, refer to the following pseudo code. This example illustrates the flow of the SYS process:

void Insert(Query query) { /// /// Insert each source record into the target. /// // connect SourceConnector.Connect(); TargetConnector.Connect(); // query source var records = SourceConnector.ExecuteQuery(query); foreach(var record in records) { // resolve formulas and map values var operationInput = BuildInsertStatement(record); // insert into target var result = TargetConnector.ExecuteOperation(operationInput); if (!result.success) { // return error LogRowError(record,result); } } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); } void InsertIfNotExists(Query query)

43

{ // // Look for each source record in the target, // insert only if a matching record is not found. // // connect SourceConnector.Connect(); TargetConnector.Connect(); // query source var records = SourceConnector.ExecuteQuery(query); foreach(var record in records) { // build target lookup query var lookupQuery = BuildLookupQuery(record); // look for matching target record var matches = TargetConnector.ExecuteQuery(lookupQuery); // insert only if no matching records are found if(matches.count == 0) { // resolve formulas and map values var operationInput = BuildInsertStatement(record); // insert into target var result = TargetConnector.ExecuteOperation(operationInput); if (!result.success) { // return error LogRowError(record,result); } } } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); } void Update(Query query) { // // For each source record, attempt to update a matching record // in the target. If the update fails because a matching record // was not found, return an error. // // connect SourceConnector.Connect(); TargetConnector.Connect(); // query source var records = SourceConnector.ExecuteQuery(query); foreach(var record in records) {

44

// resolve formulas and map values and lookup var operationInput = BuildUpdateStatement(record); // update matching target record var result = TargetConnector.ExecuteOperation(operationInput); // return an error if update did not find a record to update if(result.recordsEffected == 0) { // return error LogRowError(record,result); } else if (!result.success) { // return error LogRowError(record,result); } } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); } void UpdateIfExists(Query query) { // // For each source record, attempt to update a matching record // in the target. If the update fails because a matching record // was not found, return success. // // connect SourceConnector.Connect(); TargetConnector.Connect(); // query source var records = SourceConnector.ExecuteQuery(query); foreach(var record in records) { // resolve formulas and map values and lookup var operationInput = BuildUpdateStatement(record); // update matching target record var result = TargetConnector.ExecuteOperation(operationInput); if(!result.success) { // return error LogRowError(record,result); } } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); }

45

void UpdateThenInsert(Query query) { // // For each source record, attempt to update a matching record // in the target. If the update fails because a matching record // was not found, insert the record into the target. // // connect SourceConnector.Connect(); TargetConnector.Connect(); // query source var records = SourceConnector.ExecuteQuery(query); foreach(var record in records) { // resolve formulas and map values and lookup var operationInput = BuildUpdateStatement(record); // update matching target record var result = TargetConnector.ExecuteOperation(operationInput); // insert if update did not find a record to update if(result.recordsEffected == 0) { // resolve formulas and map values operationInput = BuildInsertStatement(record); // insert into target result = TargetConnector.ExecuteOperation(operationInput); if (!result.success) { // return error LogRowError(record,result); } } else if (!result.success) { // return error LogRowError(record,result); } } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); } void Delete(Query query) { // // For each source record, attempt to delete a matching record // in the target. If the delete fails because a matching record // was not found, return an error. // // connect

46

SourceConnector.Connect(); TargetConnector.Connect(); // query source var records = SourceConnector.ExecuteQuery(query); foreach(var record in records) { // resolve formulas and map values and lookup var operationInput = BuildDeleteStatement(record); // delete matching target record var result = TargetConnector.ExecuteOperation(operationInput); // return an error if delete did not find a record to delete if(result.recordsEffected == 0) { // return error LogRowError(record,result); } else if (!result.success) { // return error LogRowError(record,result); } } // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); } void DeleteIfExists(Query query) { // // For each source record, attempt to delete a matching record // in the target. If the delete fails because a matching record // was not found, return success. // // connect SourceConnector.Connect(); TargetConnector.Connect(); // query source var records = SourceConnector.ExecuteQuery(query); foreach(var record in records) { // resolve formulas and map values and lookup var operationInput = BuildDeleteStatement(record); // delete matching target record var result = TargetConnector.ExecuteOperation(operationInput); if(!result.success) { // return error LogRowError(record,result); }

47

} // disconnect SourceConnector.Disconnect(); TargetConnector.Disconnect(); }

48

Deploying a Connector

To deploy a connector for a Scribe Online organization, you must copy the connector .dll file to your Scribe Online Agent computer. The Agent computer automatically recognizes connector .dll files placed into subdirectories beneath the agent installation directory. By default, the Agent is installed in:

%\Program Files%\Scribe Software\Scribe Online Agent

Place each connector .dll file into a separate a folder beneath:

%\Program Files%\Scribe Software\Scribe Online Agent \Connectors\

For example, you might place a connector for Salesforce in the folder:

%\Program Files%\Scribe Software\Scribe Online Agent \Connectors\Salesforce

The Scribe Online Agent automatically recognizes new connectors when you place the connector .dll file in a subfolder. Multiple .dll files cannot share the same folder.

Note: Any .dll that you put into a folder under the Scribe Online agent’s folder and that uses the IConnector interface will be recognized by Scribe Online. Renaming a connector .dll file after it has been recognized by Scribe Online could cause unexpected results. Please take care when naming your connector, so that you do not have to change the name later.

Documents

Scribe Software CDK Developer Guide - Amazon Web Services · The API takes advantage of the latest Microsoft technologies, including .NET 4.0, Silverlight, Enterprise Service Bus,