Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The appRules Data Migration Engine is a powerful solution that has been purpose-built for data migration, data replication and data synchronization.
Powered by the appRules Platform, the appRules Data Migration Engine leverages the many built-in features of appRules to deliver a solution to successfully complete data migration, replication and synchronization projects on time and on budget.
Before exploring the features of the appRules Data Migration Engine, review the features of the appRules Platform to gain an understanding of the basic concepts and the core workflow activities used in most data migration projects.
Since appRules offers several options for data migration, replication and synchronization, it is recommended that you take the time to choose the best option for the job at hand.
While basic data migration functions can be performed using the flow data migration workflow activities, the Migrate Data Activity offers extensive powerful functionality for executing high volume data migration, replication and synchronization jobs in appRules.
The Migrate Data Activity is available in the following appRules Platform products:
You can access the Migrate Data Activity directly from the toolbox:
Platform Product | Availability |
---|---|
appRules Data Migration Engine
appRules Integration & Automation Engine
appRules Enterprise 360
To initialize the Migrate Data Activity, drag the MigrateData activity from the Migrate-Replicate-Synchronize module of the toolbox on to the designer and configure it as described in the following sections.
Select the Data Migration job type from the list.
Based on your selection of a job type, appRules will display additional dialogs to be used for setting up the values for populating the "Target Entities Collection" which is discussed in the next section.
To initialize the Migrate Data Activity from an existing activity, follow the steps below:
Select the project database and activity, then click the "Copy Properties" button.
All properties of the selected Migrate Data Activity including the "Target Entities Collection" will be copied into the current activity.
Once you have initialized and saved the Migrate Data Activity, you can edit it by clicking its Configure icon on the designer. The "Edit Activity" modal will be displayed:
Click the Copy button on the Migrate Data Activity modal to display the Copy modal.
The Target Entities Collection is the main hub of the Migrate Data Activity and is used for configuring all properties of the selected data migration job type. In this section, we will review the Target Entities Collection in general as the required entries vary based on the selected data migration job type.
Button | Description |
---|---|
Accept changes made to the target entities collection items.
Add a new collection item.
Note: All Source-Target selections are automatically included in the collection.
Remove a collection item from the list.
Make a copy of the selected collection item.
Sets a specific property value for all collection items.
Validates all collection items in the list.
Note: Property values are automatically validated.
Add, Edit, or View Conditions used in the data migration project.
Add, Edit, or View Actions used in the data migration project.
Add, Edit, or View Data Sources used in the data migration project.
Use the Data Source Browser to query and browse data records in the source or target.
Add, Edit, or View Details of the data migration project including schedule, notifications, arguments, notes and other settings.
1-Run the data migration job.
2-Rererun incomplete job or failed records.
Note: Jobs can also be run via the Scheduler, Web API or directly using the Run menu on the home page.
View the Logs and Statistics for running or completed jobs.
Use the Job Monitor to view, communicate with or terminate running jobs.
OK
Add
Remove
Copy
Set All
Validate All
Conditions
Actions
Data Sources
Data Source Browser
Project Details
Run
Logs/Statistics
Job Monitor
The "Matching Schema Migration" option is used for migrating data when the source and target share the same schema.
To perform "Matching Schema Migration" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Matching Schema Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
On the following modal, select the Target entities where the Source records will be imported.
Click the OK button to accept the entries and display the Target Entities Collection.
Since the Source and Target share a common schema, no changes are required on the Target Entities Collection. You can modify the default values to suit your needs.
Refer to the Target Entities Collection section for notes on configuring the target entities.
Run your data migration project after configuring, reviewing, and saving the Target Entities Collection.
To perform Basic Data Migration using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Basic Data Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
On the following modal, select the Target entities where the Source records will be imported.
You will be prompted for additional options for configuring the required entries for the target entities.
The Target Entities Collection modal will be displayed. Refer to the Target Entities Collection section for notes on configuring the target entities.
Run your data migration project after configuring, reviewing, and saving the Target Entities Collection.
Each Source to Target selection that you make as you define your data migration, replication or synchronization job is represented as a Target Entity definition in the Target Entities Collection.
appRules generates default values for the properties of the Target Entities. You can configure the the properties to suit your needs.
You can change the order of execution of the jobs in the Target Entities collection by simply changing the location of the items - (drag-and-drop).
Note: Jobs defined to run in parallel are not affected by the location in the collection.
Below are the properties of the Target Entity in the Target Entities Collection of the Migrate Data Activity grouped by category:
A majority of the target entity properties are automatically generated when the Migrate Data Activity is initialized.
The functionality of the Target Entity properties depends on the data migration job type. Any differences are reviewed in the section for the specific data migration job type.
If a condition is specified, the job for his Target Entity will only be run if the condition is true.
Default: None
The unique display name for the Target Entity job definition.
Default: Automatically generated
The selected data migration job type.
Whether to include the target entity in the next run of the project - (True/False).
Default: True
Use this property to configure the logging settings for the target entity. Record logging options include:
All Records
Error Records Only
No Audit Record Logging
You can also configure options to capture custom values to be stored in the logs.
Default: All Records
Optional: Configure target entities/tables that must be migrated with the current MigrationJobKey before processing the target entity.
Default: None
The option for executing the job for migrating the entity.
Default: Run the job using the value defined in the MigrateData activity.
Parallel: Run the job in parallel with other running jobs in the collection.
Sequential: Run the job sequentially - the job for the next entity on the list will not be started until the current job is completed
Default: Default
Use this property to define options for loading records including batch size, maximum number of records to load, error settings, notifications, etc.
Select the source data source to be used for the migration.
Select the primary entity of the data source to be migrated.
Select the query type to be used for loading the source records:
Configuration: Configure the source record load activity by selecting options.
CustomQuery: Use a custom query to select records.
StoredProcedure: If supported by the Source, use a stored procedure to load records.
Default: Configuration
Based on the SourceQueryType selected above, configure the settings or define a custom query for loading the source records to be migrated:
Default: None
Select the target data source to be used for the migration.
This property is automatically populated when the Migrate Data Activity is initialized.
Select the target primary entity into which the source records will be loaded.
This property is automatically populated when the Migrate Data Activity is initialized.
Optional: Configure Target Worker definitions for optimizing processing. Target Worker definitions are shared by all running target entity migration jobs initiated by the current Migrate Data Activity.
Default: Settings pre-configured.
Optional: Configure secondary entities if required for the data migration job type. For example, if the data migration job type is One-To-Many Mapping, the secondary entities are stored in this collection during the initialization of the Migrate Data Activity.
Default: None
Optional: Configure any custom actions required to initialize the job.
Default: None
Optional: Configure any custom actions to execute before mapping a data record.
Default: None
The definitions of the Source to Target field maps are automatically generated by appRules and can optionally be modified to suit your needs. For example, you can use appRules field mapping to transform Source data before saving in the Target.
Optional: Configure any custom actions to execute before saving a data record.
Default: None
Configure the settings for saving the records including Insert, Update, Upsert and Skipsert.
Default: Insert
Optional: Configure any custom actions to execute after saving a data record.
Default: None
Optional: Configure any custom actions to execute upon completion of the job for the entity.
Default: None
The "One-to-Many Migration" option is used for migrating data from a single Source data record to multiple Target data records (SecondaryEntities). The records can also be linked to create parent/child relationships.
To perform "One-to-Many Migration" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "One-to-Many Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
The "Initialize One-to-Many Migration" modal will be displayed:
Initialize the "One-to-Many Migration" option by configuring the fields as follows:
Select the Source entity/table to be migrated.
Select the primary target entity.
Select the secondary target entities.
Click the OK button to accept the entries and display the Target Entities Collection.
Note that the system automatically generates entries for the secondary entities in the Target Entities Collection.
The "Entities to Csv Migration" option is used for migrating data from a source to a folder of CSV files.
To perform "Entities to Csv Migration" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Entities to Csv Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
The following modal will be displayed:
Select the entities that will be exported to CSV.
Click the OK button to accept the entries and display the Target Entities Collection.
Since the Target is a CSV folder, most properties in the Target Entities Collection are auto-generated.
The files will be created in the default output folder: ...DbFiles[YourProjectDbName]\DataMigration\Output
The "Data Replication" option is used for replicating data from the Source to Target. The "Data Replication" option includes setting for creating entities/tables in the Target if they do not exist.
To perform "Data Replication" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Data Replication" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target from the list. The Target must be one of the supported data sources for data replication including ADLS, AmazonS3, Azure Synapse, Cassandra, DB2, Databricks, BigQuery, MySQL, Oracle, PostgreSQL, Redshift, SQL Azure, SQL Server, etc.
Click on the "Replicate In" button to configure the connection properties for the Target.
Click the OK button on the "Initialize Data Migration Activity" dialog to accept the entries and continue.
The Source Entities selection modal will be displayed:
Select the Source entities to be replicated in the Target.
Click OK and the "Target Entities Collection" modal will be displayed:
Note that the system automatically generates properties for all collection items including data record field mapping and schema for creating the target entity/table.
To view or modify the table creation properties, click on the TargetPrimaryEntityName property:
To view or modify the column definitions, click on the DataFields property:
Configure all required properties of the Target Entities Collection.
Configuration: Configure the source record load activity by configuring properties.
CustomQuery: Enter a custom SQL Query to select records from the data source.
StoredProcedure: If supported by the Source, select the stored procedure.
Optional: The optional source workers to use. Use source workers to split a very large number of source records into manageable (and distinct) groups by specifying different source queries.
Refer to the section for notes on configuring the target entities including the secondary entities.
after configuring, reviewing, and saving the .
Review the sample project to learn more about how to create "One-to-Many Migration" projects in appRules.
Refer to the section for notes on configuring the target entities.
after configuring, reviewing, and saving the .
Review the sample project to learn more about how to create "Entities to CSV Migration" projects in appRules.
Refer to the section for notes on configuring the target entities including the table creation properties.
after configuring, reviewing, and saving the .
Review the sample project to learn more about how to create a data replication project in appRules.
The "Data Synchronization" option is used for synchronizing data from the Source to Target. All changes made on the Source are applied to the Target.
To perform "Data Synchronization" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Data Synchronization" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target from the list. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button on the "Initialize Data Migration Activity" modal to accept the entries and continue.
The Target Entities selection modal will be displayed:
Select the Target entities into which Source data records will be synchronized.
Click OK and the "Target Entities Detail" modal will be displayed:
Complete the synchronization properties to be used for initializing the "Target Entities Collection" as follows:
The queries to be used to load records to be synchronized with the Target must be defined using "System Sourced Values" maintained by appRules:
To create the query for synchronizing inserts and updates for a specific entity, click the "Inserts & Updates" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime.
To create the query for synchronizing deletions for a specific entity, click the "Deletions" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime.
Once you complete the entries for data synchronization, click the OK button to display the Target Entities Collection:
Refer to the Target Entities Collection section for notes on configuring the target entities including the table creation properties.
Configure all required properties of the Target Entities Collection.
Run your data synchronization project after configuring, reviewing, and saving the Target Entities Collection.
Property | Description |
---|---|
System Sourced Value | Description |
---|---|
Target Entity Name
Enter the name of the Target entity/table into which data records from the Source will be synchronized. This property is auto-generated.
Synch Source Entity
Enter the name of the Source entity/table whose data records are to be synchronized into the Target. This property is auto-generated.
Source CreatedOn Date
The column name of datetime the record was created if supported by the source.
Source ModifiedOn Date
The column name of datetime the record was modified if supported by the source.
Auto-Map
Whether to auto-map the source data fields to the target data fields.
Synchronize Deletions
Whether to include a target entity entry to synchronize deleted records. Check to make sure that the data source supports a flag in a table for deleted records.
Inserts & Updates
Configure the query to be used for reading records that have been recently inserted or updated on the source. Note: Follow instructions to define the query to be used for loading data records to be synchronized.
Deletions
If "Synchronize Deletions" has been checked, configure the query to be used for reading records that have been recently deleted on the source. Note: Follow instructions to define query to be used for loading data records to be synchronized.
WorkflowCurrentStartDateTime
The start date/time of the currently executing instance
WorkflowLastStartDateTime
The start time of the last run of the project
The definitions of the Source to Target field maps are automatically generated by appRules and can optionally be modified to suit your needs. You can use also appRules field mapping to transform Source data before saving in the Target.
You can also optionally define Mapping Conditions to be utilized in the mapping process using the appRules Condition Editor:
To configure a Condition to be applied to a row in the MapRecordFields modal:
Configure the Condition by navigating the Condition tree on the appRules Condition Editor (Condition-->Category-->Type).
Configure the properties of the selected condition type.
Name and save the condition.
You can quickly transform values in the mapping activity modal by configuring Action Fields without writing code using the appRules Action Editor:
To transform a value using the ActionField option:
Configure the Action by navigating the Action tree on the appRules Action Editor (Field-->Field-->Module-->Type).
Configure the properties of the selected action type.
Name and save the action.
As you configure your project, appRules automatically organizes and stores the project settings. Before running the project, you can review the project settings from a central location by clicking on the Project Details button:
Review and make any changes in the following tabs:
Apply the condition by selecting it on the Mapping Condition column of a specific row of the table in the MapRecordFields modal.
On a specific row of the MapRecordFields modal, select ActionField as the Source of the value to be mapped to a target field. Then select the Action that you have created above.
Tab | Description |
---|---|
Run Options
Use this tab to make changes related to the defaults for running the job.
Status
View the current status of the project
Schedule
Use this tab to optionally schedule the job to be run
Notifications
Define optional notifications
Arguments
Define optional project variables
Activities
View the activities that have been defined for the project by dragging and dropping from the toolbox on the designer.
Notes
View or edit notes associated with the project
Json
View project JSON
Projects
View optional child projects associated with this project
The "Bidirectional Data Synchronization" option is used for synchronizing data from the Source to Target and from the Target to the Source. All changes made on the Source are applied to the Target and all changes made on the Target are applied to the Source. This is also known as two-way data synchronization.
To perform "Bidirectional Data Synchronization" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Bidirectional Data Synchronization" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target from the list. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button on the "Initialize Data Migration Activity" modal to accept the entries and continue.
The Target Entities selection modal will be displayed:
Select the Target entities into which Source data records will be synchronized.
Click OK and the "Target Entities Detail" modal will be displayed:
Complete the synchronization properties to be used for initializing the "Target Entities Collection" as follows:
The queries to be used to load records to be synchronized with the Target must be defined using "System Sourced Values" maintained by appRules:
To create the query for synchronizing inserts and updates for a specific entity, click the "Inserts & Updates" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime.
To create the query for synchronizing deletions for a specific entity, click the "Deletions" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime
Clck the OK button and the following message will be displayed:
Follow the same steps above that was used to configure Source-to-Target synchronization properties to configure Target-To-Source synchronization properties.
Once you complete the entries for bidirectional data synchronization, click the OK button to display the Target Entities Collection:
The Target Entities Collection for bidirectional data synchronization contains entries for Source-to-Target and Target-to-Source settings.
Refer to the Target Entities Collection section for notes on configuring the target entities including the table creation properties.
Configure all required properties of the Target Entities Collection.
Run your data synchronization project after configuring, reviewing, and saving the Target Entities Collection.
Property | Description |
---|---|
System Sourced Value | Description |
---|---|
Target Entity Name
Enter the name of the Target entity/table into which data records from the Source will be synchronized. This property is auto-generated.
Synch Source Entity
Enter the name of the Source entity/table whose data records are to be synchronized into the Target. This property is auto-generated.
Source CreatedOn Date
The column name of datetime the record was created if supported by the source.
Source ModifiedOn Date
The column name of datetime the record was modified if supported by the source.
Auto-Map
Whether to auto-map the source data fields to the target data fields.
Synchronize Deletions
Whether to include a target entity entry to synchronize deleted records. Check to make sure that the data source supports a flag in a table for deleted records.
Inserts & Updates
Configure the query to be used for reading records that have been recently inserted or updated on the source. Note: Follow instructions to define the query to be used for loading data records to be synchronized.
Deletions
If "Synchronize Deletions" has been checked, configure the query to be used for reading records that have been recently deleted on the source. Note: Follow instructions to define query to be used for loading data records to be synchronized.
WorkflowCurrentStartDateTime
The start date/time of the currently executing instance
WorkflowLastStartDateTime
The start time of the last run of the project
Your data migration job can be run directly, scheduled to run, or run via the appRules Web API.
To run the project via the appRules Web API, follow the instructions in Using the Web API
If you are running a long running job, you can monitor it by using the appRules Job Monitor:
View job details
Set a job control value
Pause job
Cancel job
During and after the run, you can utilize the many facilities available in appRules to view results and statistics.
You can browse the the source and newly replicated data records using the appRules Data Source Browser as shown in the figure below.
You can define queries to load records and optionally even edit data records using the appRules Data Source Browser:
To view "Data Migration Logs and Statistics", click the "Logs/Statistics" button on the "appRules Data Migration: Target Entities" toolbar. The following modal will be displayed:
appRules Data Migration Logs and Statistics offer a rich set of options to give you insight into running or completed jobs as show in the sections below. Simply configure the selection criteria that meets your requirements and click on the Summary, Statistics, Project Run Details or Audit Log Details toolbar button.
At any point, you can access the Data Migration Job Summary to view the results of running or completed instances:
Job Status
Completed Record Counts
Skipped Record Counts
Error Record Counts
Gain insight on the state of running or completed job instances by viewing job statistics.
Gain insight on the state of running or completed job instances by viewing Project Run Details:
Job Instance detail including reference, start, end, duration, current state, etc.
Snapshot View of Audit Log Details including Success, Failed, and Skipped Record counts.
Workflow Activity Statistics
Data Source Statistics
Condition Execution Results
Action Execution Results
Arguments & Variables
Runtime Logs & Errors
You can configure your appRules data migration, replication and synchronization jobs to capture all pertinent information regarding every source and target record involved in the process. This level of detail is especially useful in heavily validated organizations and offers the best tool for reconciling the results of the process.
Audit Log Details can be viewed by accessing the Data Migration Logs and Statistics modal and selecting the range of log entries to view.
Validations, error handling, and notifications are critical components for executing successful data migration, data replication and data synchronization projects.
appRules includes the following options for error handling and notifications:
Automatic and on-demand validation is available for all target entity definitions.
Click to Run toolbar button to run the project directly. Select the server and other settings for the job run.
To schedule the job, edit the Schedule tab of the Project Details:
Automatic and on-demand validation is available for all workflow activities.
Incomplete, Terminated, and errored job instances can be rerun to process only the incomplete or errored records. This saves a great deal of time as successfully transferred records are skipped - greatly reducing the load trips to the source.
Edit the LoaderSettings property of any Target Entity definition to optionally configure error tolerance threshold and the action to perform when the error limit is exceeded:
Edit the Notifications tab of the Project Settings to define project-wide notifications: