Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The Target Entities Collection is the main hub of the Migrate Data Activity and is used for configuring all properties of the selected data migration job type. In this section, we will review the Target Entities Collection in general as the required entries vary based on the selected data migration job type.
The "Data Replication" option is used for replicating data from the Source to Target. The "Data Replication" option includes setting for creating entities/tables in the Target if they do not exist.
To perform "Data Replication" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Data Replication" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target from the list. The Target must be one of the supported data sources for data replication including ADLS, AmazonS3, Azure Synapse, Cassandra, DB2, Databricks, BigQuery, MySQL, Oracle, PostgreSQL, Redshift, SQL Azure, SQL Server, etc.
Click on the "Replicate In" button to configure the connection properties for the Target.
Click the OK button on the "Initialize Data Migration Activity" dialog to accept the entries and continue.
The Source Entities selection modal will be displayed:
Select the Source entities to be replicated in the Target.
Click OK and the "Target Entities Collection" modal will be displayed:
Note that the system automatically generates properties for all collection items including data record field mapping and schema for creating the target entity/table.
To view or modify the table creation properties, click on the TargetPrimaryEntityName property:
To view or modify the column definitions, click on the DataFields property:
Refer to the Target Entities Collection section for notes on configuring the target entities including the table creation properties.
Configure all required properties of the Target Entities Collection.
Run your data migration project after configuring, reviewing, and saving the Target Entities Collection.
Review the Replicate Dynamics365 in Snowflake sample project to learn more about how to create a data replication project in appRules.
The Target Entities Collection dialog includes a toolbar for performing related activities for setting up, validating and running the data migration, replication or synchronization process.
Below are the toolbar buttons available on the modal:
Accept changes made to the target entities collection items.
Add a new collection item.
Note: All Source-Target selections are automatically included in the collection.
Remove a collection item from the list.
Make a copy of the selected collection item.
Sets a specific property value for all collection items.
Validates all collection items in the list.
Note: Property values are automatically validated.
Add, Edit, or View Conditions used in the data migration project.
Add, Edit, or View Actions used in the data migration project.
Add, Edit, or View Data Sources used in the data migration project.
Use the Data Source Browser to query and browse data records in the source or target.
Add, Edit, or View Details of the data migration project including schedule, notifications, arguments, notes and other settings.
1-Run the data migration job.
2-Rererun incomplete job or failed records.
Note: Jobs can also be run via the Scheduler, Web API or directly using the Run menu on the home page.
View the Logs and Statistics for running or completed jobs.
Use the Job Monitor to view, communicate with or terminate running jobs.
Each Source to Target selection that you make as you define your data migration, replication or synchronization job is represented as a Target Entity definition in the Target Entities Collection.
appRules generates default values for the properties of the Target Entities. You can configure the the properties to suit your needs.
You can change the order of execution of the jobs in the Target Entities collection by simply changing the location of the items - (drag-and-drop).
Note: Jobs defined to run in parallel are not affected by the location in the collection.
Below are the properties of the Target Entity in the Target Entities Collection of the Migrate Data Activity grouped by category:
A majority of the target entity properties are automatically generated when the Migrate Data Activity is initialized.
The functionality of the Target Entity properties depends on the data migration job type. Any differences are reviewed in the section for the specific data migration job type.
If a condition is specified, the job for his Target Entity will only be run if the condition is true.
Default: None
The unique display name for the Target Entity job definition.
Default: Automatically generated
The selected data migration job type.
Whether to include the target entity in the next run of the project - (True/False).
Default: True
Use this property to configure the logging settings for the target entity. Record logging options include:
All Records
Error Records Only
No Audit Record Logging
You can also configure options to capture custom values to be stored in the logs.
Default: All Records
Optional: Configure target entities/tables that must be migrated with the current MigrationJobKey before processing the target entity.
Default: None
The option for executing the job for migrating the entity.
Default: Run the job using the value defined in the MigrateData activity.
Parallel: Run the job in parallel with other running jobs in the collection.
Sequential: Run the job sequentially - the job for the next entity on the list will not be started until the current job is completed
Default: Default
Use this property to define options for loading records including batch size, maximum number of records to load, error settings, notifications, etc.
Select the source data source to be used for the migration.
Select the primary entity of the data source to be migrated.
Select the query type to be used for loading the source records:
Configuration: Configure the source record load activity by selecting options.
CustomQuery: Use a custom query to select records.
StoredProcedure: If supported by the Source, use a stored procedure to load records.
Default: Configuration
Based on the SourceQueryType selected above, configure the settings or define a custom query for loading the source records to be migrated:
Default: None
Select the target data source to be used for the migration.
This property is automatically populated when the Migrate Data Activity is initialized.
Select the target primary entity into which the source records will be loaded.
This property is automatically populated when the Migrate Data Activity is initialized.
Optional: Configure Target Worker definitions for optimizing processing. Target Worker definitions are shared by all running target entity migration jobs initiated by the current Migrate Data Activity.
Default: Settings pre-configured.
Optional: Configure secondary entities if required for the data migration job type. For example, if the data migration job type is One-To-Many Mapping, the secondary entities are stored in this collection during the initialization of the Migrate Data Activity.
Default: None
Optional: Configure any custom actions required to initialize the job.
Default: None
Optional: Configure any custom actions to execute before mapping a data record.
Default: None
The definitions of the Source to Target field maps are automatically generated by appRules and can optionally be modified to suit your needs. For example, you can use appRules field mapping to transform Source data before saving in the Target.
Optional: Configure any custom actions to execute before saving a data record.
Default: None
Configure the settings for saving the records including Insert, Update, Upsert and Skipsert.
Default: Insert
Optional: Configure any custom actions to execute after saving a data record.
Default: None
Optional: Configure any custom actions to execute upon completion of the job for the entity.
Default: None
To initialize the Migrate Data Activity, drag the MigrateData activity from the Migrate-Replicate-Synchronize module of the toolbox on to the designer and configure it as described in the following sections.
Select the Data Migration job type from the list.
To initialize the Migrate Data Activity from an existing activity, follow the steps below:
Select the project database and activity, then click the "Copy Properties" button.
Once you have initialized and saved the Migrate Data Activity, you can edit it by clicking its Configure icon on the designer. The "Edit Activity" modal will be displayed:
OK
Add
Remove
Copy
Set All
Validate All
Conditions
Actions
Data Sources
Data Source Browser
Project Details
Run
Logs/Statistics
Job Monitor
Configuration: Configure the source record load activity by configuring properties.
CustomQuery: Enter a custom SQL Query to select records from the data source.
StoredProcedure: If supported by the Source, select the stored procedure.
Optional: The optional source workers to use. Use source workers to split a very large number of source records into manageable (and distinct) groups by specifying different source queries.
Based on your selection of a job type, appRules will display additional dialogs to be used for setting up the values for populating the "which is discussed in the next section.
Click the Copy button on the Migrate Data Activity modal to display the Copy modal.
All properties of the selected Migrate Data Activity including the "will be copied into the current activity.
To perform Basic Data Migration using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Basic Data Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
On the following modal, select the Target entities where the Source records will be imported.
You will be prompted for additional options for configuring the required entries for the target entities.
The Target Entities Collection modal will be displayed. Refer to the Target Entities Collection section for notes on configuring the target entities.
Run your data migration project after configuring, reviewing, and saving the Target Entities Collection.
The "Matching Schema Migration" option is used for migrating data when the source and target share the same schema.
To perform "Matching Schema Migration" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Matching Schema Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
On the following modal, select the Target entities where the Source records will be imported.
Click the OK button to accept the entries and display the Target Entities Collection.
Since the Source and Target share a common schema, no changes are required on the Target Entities Collection. You can modify the default values to suit your needs.
Refer to the Target Entities Collection section for notes on configuring the target entities.
Run your data migration project after configuring, reviewing, and saving the Target Entities Collection.
While basic data migration functions can be performed using the flow data migration workflow activities, the Migrate Data Activity offers extensive powerful functionality for executing high volume data migration, replication and synchronization jobs in appRules.
The Migrate Data Activity is available in the following appRules Platform products:
appRules Data Migration Engine
appRules Integration & Automation Engine
appRules Enterprise 360
You can access the Migrate Data Activity directly from the toolbox:
The "Entities to Csv Migration" option is used for migrating data from a source to a folder of CSV files.
To perform "Entities to Csv Migration" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Entities to Csv Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
The following modal will be displayed:
Select the entities that will be exported to CSV.
Click the OK button to accept the entries and display the Target Entities Collection.
Since the Target is a CSV folder, most properties in the Target Entities Collection are auto-generated.
Refer to the Target Entities Collection section for notes on configuring the target entities.
Run your data migration project after configuring, reviewing, and saving the Target Entities Collection.
The files will be created in the default output folder: ...DbFiles[YourProjectDbName]\DataMigration\Output
Review the Batch Data Source Entities to Files sample project to learn more about how to create "Entities to CSV Migration" projects in appRules.
The "One-to-Many Migration" option is used for migrating data from a single Source data record to multiple Target data records (SecondaryEntities). The records can also be linked to create parent/child relationships.
To perform "One-to-Many Migration" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "One-to-Many Migration" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button to accept the entries and continue.
The "Initialize One-to-Many Migration" modal will be displayed:
Initialize the "One-to-Many Migration" option by configuring the fields as follows:
Select the Source entity/table to be migrated.
Select the primary target entity.
Select the secondary target entities.
Click the OK button to accept the entries and display the Target Entities Collection.
Note that the system automatically generates entries for the secondary entities in the Target Entities Collection.
Refer to the Target Entities Collection section for notes on configuring the target entities including the secondary entities.
Run your data migration project after configuring, reviewing, and saving the Target Entities Collection.
Review the One-to-Many Migration sample project to learn more about how to create "One-to-Many Migration" projects in appRules.
The "Data Synchronization" option is used for synchronizing data from the Source to Target. All changes made on the Source are applied to the Target.
To perform "Data Synchronization" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Data Synchronization" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target from the list. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button on the "Initialize Data Migration Activity" modal to accept the entries and continue.
The Target Entities selection modal will be displayed:
Select the Target entities into which Source data records will be synchronized.
Click OK and the "Target Entities Detail" modal will be displayed:
Complete the synchronization properties to be used for initializing the "Target Entities Collection" as follows:
The queries to be used to load records to be synchronized with the Target must be defined using "System Sourced Values" maintained by appRules:
To create the query for synchronizing inserts and updates for a specific entity, click the "Inserts & Updates" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime.
To create the query for synchronizing deletions for a specific entity, click the "Deletions" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime.
Once you complete the entries for data synchronization, click the OK button to display the Target Entities Collection:
Configure all required properties of the Target Entities Collection.
The "Bidirectional Data Synchronization" option is used for synchronizing data from the Source to Target and from the Target to the Source. All changes made on the Source are applied to the Target and all changes made on the Target are applied to the Source. This is also known as two-way data synchronization.
To perform "Bidirectional Data Synchronization" using the appRules Data Migration Engine, drag-and-drop the MigrateData workflow activity from the toolbox on to the designer and configure as follows:
Enter a display name and description for the workflow activity or accept the defaults.
Select "Bidirectional Data Synchronization" as the Data Migration Type.
Select the Default Source. If the Source data source does not exist, you can use the New Data Source button to add it.
Select the Default Target from the list. If the Target data source does not exist, you can use the New Data Source button to add it.
Click the OK button on the "Initialize Data Migration Activity" modal to accept the entries and continue.
The Target Entities selection modal will be displayed:
Select the Target entities into which Source data records will be synchronized.
Click OK and the "Target Entities Detail" modal will be displayed:
Complete the synchronization properties to be used for initializing the "Target Entities Collection" as follows:
The queries to be used to load records to be synchronized with the Target must be defined using "System Sourced Values" maintained by appRules:
To create the query for synchronizing inserts and updates for a specific entity, click the "Inserts & Updates" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime.
To create the query for synchronizing deletions for a specific entity, click the "Deletions" column on the specific row.
Note that the WorkflowLastStartDateTime and WorkflowCurrentStartDateTime have been inserted into the query.
You can modify the query to suit your needs using the toolbar buttons of the Query Editor. Use the Insert button to insert values to be generated at runtime
Clck the OK button and the following message will be displayed:
Follow the same steps above that was used to configure Source-to-Target synchronization properties to configure Target-To-Source synchronization properties.
Once you complete the entries for bidirectional data synchronization, click the OK button to display the Target Entities Collection:
The Target Entities Collection for bidirectional data synchronization contains entries for Source-to-Target and Target-to-Source settings.
Configure all required properties of the Target Entities Collection.
Refer to the section for notes on configuring the target entities including the table creation properties.
after configuring, reviewing, and saving the .
Refer to the section for notes on configuring the target entities including the table creation properties.
after configuring, reviewing, and saving the .
Target Entity Name
Enter the name of the Target entity/table into which data records from the Source will be synchronized. This property is auto-generated.
Synch Source Entity
Enter the name of the Source entity/table whose data records are to be synchronized into the Target. This property is auto-generated.
Source CreatedOn Date
The column name of datetime the record was created if supported by the source.
Source ModifiedOn Date
The column name of datetime the record was modified if supported by the source.
Auto-Map
Whether to auto-map the source data fields to the target data fields.
Synchronize Deletions
Whether to include a target entity entry to synchronize deleted records. Check to make sure that the data source supports a flag in a table for deleted records.
Inserts & Updates
Configure the query to be used for reading records that have been recently inserted or updated on the source. Note: Follow instructions to define the query to be used for loading data records to be synchronized.
Deletions
If "Synchronize Deletions" has been checked, configure the query to be used for reading records that have been recently deleted on the source. Note: Follow instructions to define query to be used for loading data records to be synchronized.
WorkflowCurrentStartDateTime
The start date/time of the currently executing instance
WorkflowLastStartDateTime
The start time of the last run of the project
Target Entity Name
Enter the name of the Target entity/table into which data records from the Source will be synchronized. This property is auto-generated.
Synch Source Entity
Enter the name of the Source entity/table whose data records are to be synchronized into the Target. This property is auto-generated.
Source CreatedOn Date
The column name of datetime the record was created if supported by the source.
Source ModifiedOn Date
The column name of datetime the record was modified if supported by the source.
Auto-Map
Whether to auto-map the source data fields to the target data fields.
Synchronize Deletions
Whether to include a target entity entry to synchronize deleted records. Check to make sure that the data source supports a flag in a table for deleted records.
Inserts & Updates
Configure the query to be used for reading records that have been recently inserted or updated on the source. Note: Follow instructions to define the query to be used for loading data records to be synchronized.
Deletions
If "Synchronize Deletions" has been checked, configure the query to be used for reading records that have been recently deleted on the source. Note: Follow instructions to define query to be used for loading data records to be synchronized.
WorkflowCurrentStartDateTime
The start date/time of the currently executing instance
WorkflowLastStartDateTime
The start time of the last run of the project
