Azure Data Lake Storage Gen2
Connecting to Azure DataLakeStorage Gen 2
To connect to a Gen 2 DataLakeStorage account, you should first set the following properties:
Schema: Set this to ADLSGen2.
Account: Set this to the name of the storage account.
FileSystem: Set this to the file system name which will be used for this account. For example, the name of an Azure Blob Container
Directory: (Optional) Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.
Authenticating to Azure DataLakeStorage Gen 2
Gen 2 supports the following authentication methods: using an AccessKey, using a Shared Access Signature, Azure Active Directory OAuth (AzureAD), Managed Service Identity (AzureMSI).
Authenticating using an Access Key
To connect using a Shared Access Signature set the AccessKey property and the AuthScheme to AccessKey.
You can obtain an access key for the ADLS Gen2 storage account using the Azure portal:
Go to your ADLS Gen2 Storage Account in the Azure portal.
Under Settings, select Access keys.
Copy the value for one of the available access keys to the AccessKey connection property.
Shared Access Signature (SAS)
To connect using a Shared Access Signature set the SharedAccessSignature property to a valid signature of a resource to connect to and the AuthScheme to SAS. The SharedAccessSignature may be generated with a tool such as Azure Storage Explorer.
Azure AD
Azure AD is a connection type that leverages OAuth to authenticate. Set your AuthScheme to AzureAD. The rest of the AzureAD flows assume that you have done so.
Custom AzureAD App
See Creating a Custom AzureAD App for information about creating custom applications and reasons for doing so.
After setting the following connection properties, you are ready to connect:
InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
OAuthClientId: (custom applications only) Set this to the client Id in your application settings.
OAuthClientSecret: (custom applications only) Set this to the client secret in your application settings.
CallbackURL: Set this to the Redirect URL in your application settings.
When you connect the provider opens the OAuth endpoint in your default browser. Log in and grant permissions to the application. The provider then completes the OAuth process:
Extracts the access token from the callback URL and authenticates requests.
Obtains a new access token when the old one expires.
Saves OAuth values in OAuthSettingsLocation that persist across connections.
Admin Consent
Admin consent refers to when the Admin for an Azure Active Directory tenant grants permissions to an application which requires an admin to consent to the use case.
Admin Consent Permissions
When creating a new OAuth app in the Azure Portal, you must specify which permissions the app will require. Some permissions may be marked stating "Admin Consent Required". For example, all Groups permissions require Admin Consent. If your app requires admin consent, there are a couple of ways this can be done.
The easiest way to grant admin consent is to just have an admin log into portal.azure.com and navigate to the app you have created in App Registrations. Under API Permissions, there will be a button for Grant Consent. You can consent here for your app to have permissions on the tenant it was created under.
Once an admin grants consent, authentication may be performed as normal.
Client Credentials
Client credentials refers to a flow in OAuth where there is no direct user authentication taking place. Instead, credentials are created for just the app itself. All tasks taken by the app are done without a default user context. This makes the authentication flow a bit different from standard.
Client OAuth Flow
In your App Registration in portal.azure.com, navigate to API Permissions and select the Microsoft Graph permissions. There are two distinct sets of permissions - Delegated and Application permissions. The permissions used during client credential authentication are under Application Permissions. Select the applicable permissions you require for your integration.
You are ready to connect after setting one of the below connection properties groups depending on the authentication type.
Client Secret
InitiateOAuth: Set this to GETANDREFRESH. You can cuse InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
AzureTenant: Set this to the tenant you wish to connet to.
OAuthGrantType: Set this to CLIENT.
OAuthClientId: Set this to the Client Id in your app settings.
OAuthClientSecret: Set this to the Client Secret in your app settings.
Certificate
InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
AzureTenant: Set this to the tenant you wish to connect to.
OAuthGrantType: Set this to CLIENT.
OAuthClientId: Set this to the Client Id in your app settings.
OAuthJWTCert: Set this to the JWT Certificate store.
OAuthJWTCertType: Set this to the type of the certificate store specified by OAuthJWTCert.
Authentication with client credentials will take place automatically like any other connection, except there will be no window opened prompting the user. Because there is no user context, there is no need for a browser popup. Connections will take place and be handled internally.
Azure Service Principal
Azure Service Principal is a connection type that goes through OAuth. Set your AuthScheme to AzureServicePrincipal. The authentication as an Azure Service Principal is handled via the OAuth Client Credentials flow, and it does not involve direct user authentication. Instead, credentials are created for just the app itself. All tasks taken by the app are done without a default user context, but based on the assigned roles. The application access to the resources is controlled through the assigned roles' permissions.
When authenticating using an Azure Service Principal, you must register an application with an Azure AD tenant. Follow the steps below to create a new service principal that can be used with the role-based access control.
Assign a role to the application
To access resources in your subscription, you must assign a role to the application.
Open the Subscriptions page by searching and selecting the Subscriptions service from the search bar.
Select the particular subscription to assign the application to.
Open the Access control (IAM) and select Add > Add role assignment to open the Add role assignment page.
Select Owner as the role to assign to your created Azure AD app.
Complete the Authentication
You are ready to connect after setting one of the below connection properties groups, depending on the configured app authentication (client secret or certificate).
In both methods
Before choosing client secret or certicate authentication, follow these steps then continue to the relevant section below:
AuthScheme: Set this to the AzureServicePrincipal in your app settings.
InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
AzureTenant: Set this to the tenant you wish to connect to.
OAuthClientId: Set this to the client Id in your app settings.
Authenticating using a Client Secret
Continue with the following:
OAuthClientId: Set this to the client Id in your app settings.
OAuthClientSecret: Set this to the client secret in your app settings.
Authenticating using a Certificate
Continue with the following:
OAuthJWTCert: Set this to the JWT Certificate store.
OAuthJWTCertType: Set this to the type of the certificate store specified by OAuthJWTCert.
MSI
If you are running Azure Data Lake Storage on an Azure VM, you can leverage Managed Service Identity (MSI) credentials to connect:
AuthScheme: Set this to AzureMSI.
The MSI credentials are automatically obtained for authentication.
Last updated