appRules features an unmatched number of options for migrating, replicating and synchronizing data.

Product & Hosting Options

Platform Product Availability

The workflow activities and other modules used in this sample project are available in the following appRules Platform products:

Hosting & Execution Environment

If you are running the On-Premise, Azure Marketplace or AWS Marketplace editions, the following hosting options supported by appRules are recommended for running this sample project:

This sample project has been configured to show how to use a regular flow to export records from a SQL database to a delimited file (CSV)

For this project, you must add the BaseData module to the toolbox. The BaseData module is used for generically accessing data sources. The same activities can be accessed from their specific modules. For example, the SQL initialization/access activities in this sample project can be accessed from the EmbeddedDB module and the CSV initialization/access activities can be accessed from the DelimitedFile module.

Activities:

1. InitializeSource – Initialize SQL Source

This activity initializes the SQL source. Check out the properties of this activity to see how you select fields, define filters etc. for reading the data from a data source.

2. InitializeTargetFile – Initialize CSV Target File

This activity initializes the CSV target file. Check out the properties of this activity to see how you select fields etc. for saving data into a csv file.

3. While Activity

This While activity has been configured for iterating the records in the SQL source

• MapAndSetFields

This activity maps and sets the CSV record field values.

• WriteDelimitedRecord

This activity writes a record in the delimited file.

• GetNextRecord

This activity reads the next record from the SQL data source

4. CloseSource

This activity closes the SQL source and releases resources. Note: This is also done automatically if the CloseSource activity was not included.

5. CloseTarget

This activity closes the CSV file and releases resources. Note: This is also done automatically if the CloseTarget activity was not included.

appRules features an unmatched number of options for migrating, replicating and synchronizing data.

appRules Data Replication

The "Replicate Dynamics365 In Snowflake" sample project shows how to utilize the appRules Data Migration Engine for enterprise-scale data replication. As you review the project, note the following:

• The entire project uses a single workflow activity - the MigrateData activity.

• Configurations are used for all aspects of the project instead of scripts or code.

• Easily configurable options for optimizing processing including Source and Target workers.

• The process is the same not only for Dynamics365 but for hundreds of sources, such as Dynamics CRM, Salesforce, SharePoint, etc.

• The process is the same not only for Snowflake but for other supported targets of appRules Data Replication, such as ADLS, AmazonS3, Azure Synapse, Cassandra, DB2, Databricks, BigQuery, MySQL, Oracle, PostgreSQL, Redshift, SQL Azure, SQL Server, etc.

Product & Hosting Options

Platform Product Availability

The workflow activities and other modules used in this sample project are available in the following appRules Platform products:

Hosting & Execution Environment

If you are running the On-Premise, Azure Marketplace or AWS Marketplace editions, the following hosting options supported by appRules are recommended for running this sample project:

Creating A New Project

First, we will create a new project for the data replication job. Click the New button on the Project toolbar of the appRules Home Page. Simply enter the name of the project - (Replicate Dynamics365 In Snowflake) and click the Save button.

The MigrateData Activity

We will configure the entire job using the appRules MigrateData activity. Drag the MigrateData activity from the Migrate-Replicate-Synchronize module of the toolbox on to the designer and configure it as described in the following sections.

Dynamics365 Connection & Metadata

appRules uses data source metadata extensively for all processing. Before configuring the migration project we must first connect to Dynamics365 to generate the metadata of the data source. Once generated, the data source metadata can be used in all your appRules projects including Business Rules, Integration and Automation. There is no limit on the number of data sources and related metadata.

To generate metadata and create the Dynamics365 data source, click the "New DataSource" button on the toolbar. Select the Connector (from CRM or ERP categories) and configure as shown in figures below. Follow subsequent prompts to generate the metadata.

Initializing the Data Replication Process

Now that we have created the Dynamics365 data source, select Dynamics365 as the default source, Data Replication as the data migration type, and Snowflake as the target. On subsequent modals, click the OK toolbar button to continue.

Snowflake Connection Properties

Configure the connection properties and schema options for Snowflake:

Dynamics365 Source Entities

Select the Dynamics365 entities to be replicated in Snowflake:

Snowflake Target Entities & Tables

The entities to be replicated are shown in the appRules Data Replication Target Entities collection:

Snowflake Table Definitions

The definitions of the target tables are automatically generated by appRules and can be modified to suit your needs:

Snowflake Table Column Definitions

The definitions of the target table columns are automatically generated by appRules and can be modified to suit your needs:

Field Mapping & Transformation

The definitions of the Dynamics365 to Snowflake field maps are automatically generated by appRules and can optionally be modified to suit your needs. For example, you can use appRules field mapping to transform Dynamic365 data before saving in Snowflake.

The MapRecordFields Activity

Conditional Field Mappings

You can also optionally define Mapping Conditions to be utilized in the mapping process using the appRules Condition Editor:

To configure a Condition to be applied to a row in the MapRecordFields modal:

• Configure the Condition by navigating the Condition tree on the appRules Condition Editor (Condition-->Category-->Type).

• Configure the properties of the selected condition type.

• Name and save the condition.

Transformations Using Action Fields

You can quickly transform values in the mapping activity modal by configuring Action Fields without writing code using the appRules Action Editor:

To transform a value using the ActionField option:

• Configure the Action by navigating the Action tree on the appRules Action Editor (Field-->Field-->Module-->Type).

• Configure the properties of the selected action type.

• Name and save the action.

Other Target Entity Properties

The appRules Data Replication Target Entities collection is automatically generated for you and is ready to run the replication jobs for the selected entities. As an option, you can make changes to the entities to suit your needs:

• Job Execution Mode - Select Default, Sequential or Parallel execution

• More ... Explore the other options of the collection entries for the entities to be replicated

Reviewing the Project Settings

As you configure your project, appRules automatically organizes and stores the project settings. Before running the project, you can review the project settings from a central location by clicking on the Project Details button:

Review and make any changes in the following tabs:

• Run Options: Use this tab to make changes related to the defaults for running the job.

• Status: View the current status of the project

• Schedule: Use this tab to optionally schedule the job to be run

• Notifications: Define optional notifications

• Arguments: Define optional project variables

• Activities: View the activities that have been defined for the project by dragging and dropping from the toolbox on the designer.

• Notes: View or edit notes associated with the project

• Json: View project JSON

• Projects: View optional child projects associated with this project

Validations, Error Handling & Notifications

Validations, error handling, and notifications are critical components for executing successful data migration, data replication and data synchronization projects.

appRules includes the following options for error handling and notifications:

• Automatic and on-demand validation is available for all target entity definitions.

Job Execution & Monitoring

Configure each entity replication entry to suit your needs. Then the job can be run directly, scheduled to run, or run via the appRules Web API.

Job Execution Options

• To run the project via the appRules Web API, follow the instructions in Using the Web API

The appRules Job Monitor

If you are running a long running job, you can monitor it by using the appRules Job Monitor:

• View job details

• Set a job control value

• Pause job

• Cancel job

Evaluating the Job Results

During and after the run, you can utilize the many facilities available in appRules to view results and statistics.

The appRules Data Source Browser

You can browse the the source and newly replicated data records using the appRules Data Source Browser as shown in the figure below.

You can define queries to load records and optionally even edit data records using the appRules Data Source Browser:

Logs and Statistics

To view "Data Migration Logs and Statistics", click the "Logs/Statistics" button on the "appRules Data Migration: Target Entities" toolbar. The following modal will be displayed:

appRules Data Migration Logs and Statistics offer a rich set of options to give you insight into running or completed jobs as show in the sections below. Simply configure the selection criteria that meets your requirements and click on the Summary, Statistics, Project Run Details or Audit Log Details toolbar button.

Job Summary

At any point, you can access the Data Migration Job Summary to view the results of running or completed instances:

• Job Status

• Completed Record Counts

• Skipped Record Counts

• Error Record Counts

Job Statistics

Gain insight on the state of running or completed job instances by viewing job statistics.

Project Run Details

Gain insight on the state of running or completed job instances by viewing Project Run Details:

• Job Instance detail including reference, start, end, duration, current state, etc.

• Snapshot View of Audit Log Details including Success, Failed, and Skipped Record counts.

• Workflow Activity Statistics

• Data Source Statistics

• Condition Execution Results

• Action Execution Results

• Arguments & Variables

• Runtime Logs & Errors

Audit Log Details

You can configure your appRules data migration, replication and synchronization jobs to capture all pertinent information regarding every source and target record involved in the process. This level of detail is especially useful in heavily validated organizations and offers the best tool for reconciling the results of the process.

Audit Log Details can be viewed by accessing the Data Migration Logs and Statistics modal and selecting the range of log entries to view.

Conclusions and Next Steps

Conclusions

Powered by the all-in-one appRules Platform, the appRules Data Migration Engine offers the most powerful and flexible end-to-end solution for enterprise-scale data migration, data replication and data synchronization for IT Departments, Solution Providers, and Consultants.

This sample project has been used to showcase the powerful, easy-to-use, and fully configurable features of appRules including:

• No-code environment

• Replicates data from hundreds of sources including SaaS platforms into popular targets including SQL databases and data warehouses such as ADLS, AmazonS3, Azure Synapse, Cassandra, DB2, Databricks, BigQuery, MySQL, Oracle, PostgreSQL, Redshift, SQL Azure, SQL Server, etc.

• Integrated support for hundreds of data source connectors.

• Metadata-driven solution

• All settings including source entities, target entities collection, target tables, and field mappings are automatically generated and can optionally be modified for customized implementations without writing code.

• Conditioned field mappings using the powerful appRules Condition Editor.

• Field transformations using Action Fields configured using the powerful appRules Action Editor.

• Integrated Job Monitors, Logs, Statistics and the appRules Data Source Browser.

Next Steps

• You can continue exploring the many features available for data migration, data replication and data synchronization in the appRules Data Migration engine. The completed project is available in the appRules Samples project database. Click the "Open" toolbar button on the appRules home page to select the "Replicate Dynamics365 In Snowflake" project.

• If you require assistance, please contact support@appstrategy.com

appRules features an unmatched number of options for migrating, replicating and synchronizing data.

appRules Parent/Child Migrations

The "Migrate Parent/Child Records Using Target/Bulk Loader" sample project demonstrates another option for executing parent/child migrations and features the following:

• The InitializeTarget workflow activity activated for Bulk Loading

• The MapRecordFields activity used for mapping, inserting records and record iteration

Product & Hosting Options

Platform Product Availability

The workflow activities and other modules used in this sample project are available in the following appRules Platform products:

Hosting & Execution Environment

If you are running the On-Premise, Azure Marketplace or AWS Marketplace editions, the following hosting options supported by appRules are recommended for running this sample project:

This sample project demonstrates how to use only a few activities to migrate parent/child records.

Workflow Activities

InitializeSource - SQL Database Customers

Reads records from the Customers table of the SQL database to be imported into the Salesforce Account and Contact objects.

Salesforce.InitializeTarget - Salesforce.Account

Initialize target for the Salesforce Account records. Customer company records will be stored using this activity.

Salesforce.InitializeTarget - Salesforce.Contact

Initialize target for the Salesforce Contact records. Customer contact records will be stored using this activity.

While Loop Activity

The while loop activity is used for iterating the records in the Customer table and mapping accounts/contacts, and specifying the Save options. It contains the two MapRecordFields workflow activities used in the project.

MapRecordFields - Map-Save Account

Maps Customer record using company fields to Salesforce Account record. After mapping, the record is saved using this activity.

MapRecordFields - Map-Save Contact

Maps Customer record using company fields to Salesforce Account record. After mapping, the record is saved using this activity

appRules features an unmatched number of options for migrating, replicating and synchronizing data.

Product & Hosting Options

Platform Product Availability

The workflow activities and other modules used in this sample project are available in the following appRules Platform products:

Hosting & Execution Environment

If you are running the On-Premise, Azure Marketplace or AWS Marketplace editions, the following hosting options supported by appRules are recommended for running this sample project:

The One-to-Many Data Migration sample project demonstrates how to use the appRules MigrateData activity to perform One-to-Many data migrations.

In the sample project, a single activity is configured to load the Customers table from the SQL Database into the Account and Contact tables in Salesforce.

Drag the MigrateData activity from the Migrate-Replicate-Synchronize module of the toolbox on to the designer and configure it as follows:

Select the default Source, Data Migration Type, and default Target:

Configure the Source Entity, the Primary Target Entity, and Secondary Target Entities:

Complete the configuration of the Target Entities to suit your needs including mapping, etc:

Notice the entries in the SecondaryEntities collection. Configure to suit your needs - mapping etc:

After running the project, you can view the Project Details and Audit Logs records and statistics.

appRules features an unmatched number of options for migrating, replicating and synchronizing data.

Product & Hosting Options

Platform Product Availability

The workflow activities and other modules used in this sample project are available in the following appRules Platform products:

Hosting & Execution Environment

If you are running the On-Premise, Azure Marketplace or AWS Marketplace editions, the following hosting options supported by appRules are recommended for running this sample project:

The '"Batch Data Source Entities to Files" sample project demonstrates how to use the appRules MigrateData activity to perform batch migration of entities in a data source to files in a folder.

In the sample project, a single workflow activity is configured to load tables from a SQL Database into CSV files in a folder.

Drag the MigrateData activity from the Migrate-Replicate-Synchronize module of the toolbox on to the designer and configure it as follows:

Select Source Database as the default source, and Export to Files as the data migration type.

Select the tables to be exported:

All properties required for the Target Entities collection will be automatically set for the process. You can make any required changes to suit your needs.

Run the project and the files will be created in the Output folder:

...DbFiles\Samples\DataMigration\Output\Source Database-Files