Azure Data Lake Storage Gen1

Connecting to Azure DataLakeStorage Gen 1

To connect to a Gen 1 DataLakeStorage account, you should first set the following properties:

  • Schema: Set this to ADLSGen1.

  • Account: Set this to the name of the account.

  • AzureTenant: Set this to the tenant Id. See the property for more information on how to acquire this.

  • Directory: (Optional) Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.

Authenticating to Azure DataLakeStorage Gen 1

Gen 1 supports the following authentication methods: Azure Active Directory OAuth (AzureAD) and Managed Service Identity (AzureMSI).

Azure AD

Azure AD is a connection type that leverages OAuth to authenticate. OAuth requires the authenticating user to interact with Azure Data Lake Storage using an internet browser. The provider facilitates this in several ways as described below. Set your AuthScheme to AzureAD. The rest of the AzureAD flows assume that you have done so.

Custom Azure oAuth App

See Creating a Custom AzureAD App for information about creating custom applications and reasons for doing so.

After setting the following connection properties, you are ready to connect:

  • InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.

  • OAuthClientId: (custom applications only) Set this to the client Id in your application settings.

  • OAuthClientSecret: (custom applications only) Set this to the client secret in your application settings.

  • CallbackURL: Set this to the Redirect URL in your application settings.

When you connect the provider opens the OAuth endpoint in your default browser. Log in and grant permissions to the application. The provider then completes the OAuth process:

  1. Extracts the access token from the callback URL and authenticates requests.

  2. Obtains a new access token when the old one expires.

  3. Saves OAuth values in OAuthSettingsLocation that persist across connections.

Admin Consent

Admin consent refers to when the Admin for an Azure Active Directory tenant grants permissions to an application which requires an admin to consent to the use case.

Admin Consent Permissions

When creating a new OAuth app in the Azure Portal, you must specify which permissions the app will require. Some permissions may be marked stating "Admin Consent Required". For example, all Groups permissions require Admin Consent. If your app requires admin consent, there are a couple of ways this can be done.

The easiest way to grant admin consent is to just have an admin log into portal.azure.com and navigate to the app you have created in App Registrations. Under API Permissions, there will be a button for Grant Consent. You can consent here for your app to have permissions on the tenant it was created under.

Once an admin grants consent, authentication may be performed as normal.

Client Credentials

Client credentials refers to a flow in OAuth where there is no direct user authentication taking place. Instead, credentials are created for just the app itself. All tasks taken by the app are done without a default user context. This makes the authentication flow a bit different from standard.

Client OAuth Flow

All permissions related to the client oauth flow require admin consent.

In your App Registration in portal.azure.com, navigate to API Permissions and select the Microsoft Graph permissions. There are two distinct sets of permissions - Delegated and Application permissions. The permissions used during client credential authentication are under Application Permissions. Select the applicable permissions you require for your integration.

You are ready to connect after setting one of the below connection properties groups depending on the authentication type.

  1. Client Secret

    • InitiateOAuth: Set this to GETANDREFRESH. You can cuse InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.

    • AzureTenant: Set this to the tenant you wish to connet to.

    • OAuthGrantType: Set this to CLIENT.

    • OAuthClientId: Set this to the Client Id in your app settings.

    • OAuthClientSecret: Set this to the Client Secret in your app settings.

  2. Certificate

    • InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.

    • AzureTenant: Set this to the tenant you wish to connect to.

    • OAuthGrantType: Set this to CLIENT.

    • OAuthClientId: Set this to the Client Id in your app settings.

    • OAuthJWTCert: Set this to the JWT Certificate store.

    • OAuthJWTCertType: Set this to the type of the certificate store specified by OAuthJWTCert.

Authentication with client credentials will take place automatically like any other connection, except there will be no window opened prompting the user. Because there is no user context, there is no need for a browser popup. Connections will take place and be handled internally.

Azure Service Principal

Azure Service Principal is a connection type that goes through OAuth. Set your AuthScheme to AzureServicePrincipal. The authentication as an Azure Service Principal is handled via the OAuth Client Credentials flow, and it does not involve direct user authentication. Instead, credentials are created for just the app itself. All tasks taken by the app are done without a default user context, but based on the assigned roles. The application access to the resources is controlled through the assigned roles' permissions.

When authenticating using an Azure Service Principal, you must register an application with an Azure AD tenant. Follow the steps below to create a new service principal that can be used with the role-based access control.

Assign a role to the application

To access resources in your subscription, you must assign a role to the application.

  1. Open the Subscriptions page by searching and selecting the Subscriptions service from the search bar.

  2. Select the particular subscription to assign the application to.

  3. Open the Access control (IAM) and select Add > Add role assignment to open the Add role assignment page.

  4. Select Owner as the role to assign to your created Azure AD app.

Complete the Authentication

You are ready to connect after setting one of the below connection properties groups, depending on the configured app authentication (client secret or certificate).

In both methods

Before choosing client secret or certicate authentication, follow these steps then continue to the relevant section below:

  1. AuthScheme: Set this to the AzureServicePrincipal in your app settings.

  2. InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.

  3. AzureTenant: Set this to the tenant you wish to connect to.

  4. OAuthClientId: Set this to the client Id in your app settings.

Authenticating using a Client Secret

Continue with the following:

  1. OAuthClientId: Set this to the client Id in your app settings.

  2. OAuthClientSecret: Set this to the client secret in your app settings.

Authenticating using a Certificate

Continue with the following:

  1. OAuthJWTCert: Set this to the JWT Certificate store.

  2. OAuthJWTCertType: Set this to the type of the certificate store specified by OAuthJWTCert.

MSI

If you are running Azure Data Lake Storage on an Azure VM, you can leverage Managed Service Identity (MSI) credentials to connect:

  • AuthScheme: Set this to AzureMSI.

The MSI credentials are automatically obtained for authentication.

PreviousData WarehouseNextAzure Data Lake Storage Gen2

Last updated