Handle Secrets With .NET Core

Have you ever checked in a configuration file with a secret? Now is the time for you to do something better. In this post, I will show four convenient ways in which configuration secrets can be stored outside of your source code repository.

TL;DR

NuGet-package Microsoft.Extensions.Configuration.EnvironmentVariables:
1
2
3
string secret = builder
    .AddEnvironmentVariables(prefix)
    .Build()[key];
NuGet-package Microsoft.Extensions.Configuration.UserSecrets:
1
2
3
string secret = builder
    .AddUserSecrets(userSecretsId)
    .Build()[key];

File with secrets at %APPDATA%\Microsoft\UserSecrets\{userSecretsId}\secrets.json.

NuGet-package Microsoft.Extensions.Configuration.AzureKeyVault:
1
2
3
string secret = builder
    .AddAzureKeyVault(vaultUrl, clientId, certificate)
    .Build()[key];
NuGet-packages Microsoft.Azure.KeyVault and Microsoft.Azure.Services.AppAuthentication:
1
2
3
var secret = await new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(
    new AzureServiceTokenProvider().KeyVaultTokenCallback))
    .GetSecretAsync(vaultUrl, key);

Secrets Stored Locally

Imagine that you are part of a team and that you work on developing some new system. Since the system depends on some back-end service you have a password in the systems .config- or .json configuration file, if not for something else, just so that you can to run the system on your local machine.

When you commit your code, you couple the access restrictions of the beck-end service to that of the repository. Or in other words, you will have to make sure that no one has access to the source code that is not allowed to access the back-end service.

Yes, you can always remove the password from the repository history, or even rotate the secret on the back-end service when development on your new system has stopped. But those two options might show to be quite cumbersome in practice.

A better option would have been to store the password on a location outside of your source code repository.

Environment Variables

Using environment variables to store secrets is a simple but effective solution. Since environment variables is a standardized mechanism that is available in most operating systems, there is a wide range of tooling that can make your life easier. One example is Docker which has great support for setting environment variables for containers.

Here is an example of how to use the AddEnvironmentVariables extension method of the NuGet package Microsoft.Extensions.Configuration.EnvironmentVariables.

1
2
3
4
5
6
7
8
9
10
public static class Program
{
    private static void Main()
    {
        var builder = new ConfigurationBuilder();
        builder.AddEnvironmentVariables("MYAPP_");
        IConfiguration configuration = builder.Build();
        Console.WriteLine($"Ex1 EnvironmentVariables: {configuration["MySecret"]}");
    }
}

In my opinion, it is a good practice to use a prefix filter. In the myriad of other environment variables, I find it nice to have the ones belonging to my apps grouped together. In my example above, the value of the variable with name MYAPP_MYSECRET will be available through configuration["MySecret"].

Also, a good feature is to group related variables into sections. When environment variables are added to the configuration instance, names with __ are replaced with :. This enables getting sub sections, or/and using the options functionality.

If you are used to get configuration values through a static resource such as the ConfigurationManager of the .NET ?Full? Framework, I recommend reading about how dependency injection can be done with ASP.NET Core, just to get you started with how to inject the configuration instance to where it is needed.

User Secrets

Another option is to use the AddUserSecrets extension method of the NuGet package Microsoft.Extensions.Configuration.UserSecrets. It enforces that the secrets are stored in the AppData- or Home-directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public static class Program
{
    private static void Main()
    {
        var builder = new ConfigurationBuilder();
        var env = new HostingEnvironment { EnvironmentName = EnvironmentName.Development };
        if (env.IsDevelopment())
        {
            builder.AddUserSecrets("MyUserSecretsId");
        }
        IConfiguration configuration = builder.Build();
        Console.WriteLine($"Ex2 UserSecrets: {configuration["MySecret"]}");
    }
}

Secrets are read from a json-file located at %APPDATA%\Microsoft\UserSecrets\{UserSecretsId}\secrets.json on Windows or ~/.microsoft/usersecrets/{UserSecretsId}/secrets.json on Linux. I find it most convenient to supply the application id directly in the AddUserSecrets-method, but you can set the id in the .csproj-file as well, or by using the assembly attribute like [assembly:UserSecretsId("MyUserSecrets")]. If you really want to do something hard core, you can set the UserSecretsId with a MSBuild property like /p:UserSecretsId=MyUserSecrets.

There is some CLI tooling ment help you manage the secrets.json file. But I find it so dead simple to create the secrets.json file manually that I seldom bother to use the CLI.

The content of secrests.json can be something like this:

1
2
3
{
  "MySecret": "Some value"
}

Since it is not a good practice to have unencrypted secrets on the file system, I recommend only using UserSecrets for local development.

Secrets Stored in a Shared Location

Back to my imaginary development scenario.

All is working well, and every team member is productive running the system. Your team is making great progress. But then all of a sudden, the back-end service password is rotated for whatever reason, and productivity stops.

Of course, one could prioritize to take the time to prepare everyone in the team of the rotation. But if you don?t, your colleagues will get runtime exceptions when they run their system. When they have found out that the reason that they got access denied was because of a rotated password, they can finally get up to speed again. That is, if they are able to find out what the new password is.

A possible way to solve this is to store the password in a shared location such in a common database, which preferably is encrypted. That is a solution that works well, and it is normally not that hard to encrypt the passwords. At least not with SQL Server.

Azure Key Vault

An even better approach would be to use a product specialized for secrets distribution, such as Azure Key Vault (AKV). I think the tooling around AKV is great. Since it has a REST API, you can access it from most platforms, and there is support in Azure Arm Templates to get secrets from AKV when they are run. Besides usability, AKV is not very expensive to use for secrets. It is so cheap that I mostly consider it to be a free service.

For accessing secrets in AKV one needs to authenticate and pass in an access token. Applications that need to authenticate with Azure Active Directory (AAD) do so with credentials stored in service principals. If needed, an application can have many ways to authenticate, and each set of credentials is stored in a separate service principal.

Authentication with service principals are made with an application id and either a client secret or a certificate. So, which one is the better option?

Secrets might seem easy to use but have the drawback that they can be read. By using secrets, you again couple the access restrictions. This time, the coupling is between access to the settings of the application and to whatever resources that application is meant to access.

Certificates are in fact also relatively straight-forward to use. You can even generate a self-signed certificate and use that to create a service principal in AAD with just these few lines of PowerShell:

1
2
3
4
5
6
7
8
9
$cert = New-SelfSignedCertificate -CertStoreLocation "cert:\CurrentUser\My" `
  -Subject "CN=my-application" -KeySpec KeyExchange
$keyValue = [System.Convert]::ToBase64String($cert.GetRawCertData())

$sp = New-AzureRMADServicePrincipal -DisplayName my-application `
  -CertValue $keyValue -EndDate $cert.NotAfter -StartDate $cert.NotBefore
Sleep 20 # Wait for service principal to be propagated throughout AAD
New-AzureRmRoleAssignment -RoleDefinitionName Contributor `
  -ServicePrincipalName $sp.ApplicationId

If you want a longer certificate validity than one year, use the argument -NotAfter for New-SelfSignedCertificate.

Here is an example of how to get a secret from AKV by using the AddAzureKeyVault extension method of the NuGet package Microsoft.Extensions.Configuration.AzureKeyVault:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
public static class Program
{
    private static void Main()
    {
        var builder = new ConfigurationBuilder();
        builder.AddAzureKeyVault(
            vault: "https://my-application-kv.vault.azure.net/",
            clientId: "865f36f7-08c1-4ca2-97c9-a5a9cab56fd8",
            certificate: GetCertificate());
        IConfiguration configuration = builder.Build();
        Console.WriteLine($"Ex3 AzureKeyVault: {configuration["MySecret"]}");
    }

    private static X509Certificate2 GetCertificate()
    {
        using (X509Store store = new X509Store(StoreLocation.CurrentUser))
        {
            store.Open(OpenFlags.ReadOnly);
            var cers = store.Certificates.Find(
                X509FindType.FindBySubjectName, "my-application", false);
            if (cers.Count == 0)
                throw new Exception("Could not find certificate!");
            return cers[0];                
        }
    }
}

For loading a certificate in an Azure App Service, add a WEBSITE_LOAD_CERTIFICATES app setting with the certificate thumbprint as value. For more details on this, read the official docs.

When calling the Build-method, all secrets are read in from AKV at once and are then kept in the configuration. Secret names with -- are replaced with : when they are read in.

AppAuthentication and KeyVaultClient

Just a few days ago, Microsoft released the NuGet package Microsoft.Azure.Services.AppAuthentication. It contains an AzureServiceTokenProvider that abstracts how to get an access token from AAD.
To use it together with the KeyVaultClient in the NuGet package Microsoft.Azure.KeyVault, you simply insert a callback method of the token provider in its constructor.

public static class Program
{
    private static async Task Main()
    {
        var azureServiceTokenProvider = new AzureServiceTokenProvider();
        var keyVaultClient = new KeyVaultClient(
            new KeyVaultClient.AuthenticationCallback(
                azureServiceTokenProvider.KeyVaultTokenCallback));
        var secret = await keyVaultClient
            .GetSecretAsync(
                vaultBaseUrl: "https://my-application-kv.vault.azure.net/",
                secretName: "MySecret");        
        Console.WriteLine($"Ex4 AppAuthentication: {secret.Value}");
    }
}

To configure how AzureServiceTokenProvider will acquire tokens you can provide a connection string, either by passing it as a parameter in the constructor or by setting it as the environment variable AzureServicesAuthConnectionString. If no connection string is provided, such as in my example above, the AzureServiceTokenProvider will try three connection strings for you and pick one that works.

If you are using the NuGet package Microsoft.Extensions.Configuration.AzureKeyVault to get secrets from AKV, which I wrote about in the previous example, you need to have a service principal for local development, preferably with a certificate for authentication. Yes, you can use Microsoft.Extensions.Configuration.EnvironmentVariables or Microsoft.Extensions.Configuration.UserSecrets to add secrets. But as I mentioned before, distributing them in your team might be an unnecessary pain.

AzureServiceTokenProvider solves this rather nice. You and your team members can use your own accounts for accessing AKV, and your production application can use its own account for accessing AKV. Everything is either picked out for you, or it can be configured at deploy-time.

I have tried to summarize the supported connection string types below for you. For full details see the official documentation.

Local development

For local development scenarios, credentials can be taken from a live Azure CLI session, a logged in user in Visual Studio, or the local user account on a computer that is joined to the domain of the AKV.

  • RunAs=Developer; DeveloperTool=AzureCli
  • RunAs=Developer; DeveloperTool=VisualStudio
  • RunAs=CurrentUser;
Since Visual Studio 2017 Update 6 you can set the account under Tools -> Azure Service Authentication.

If you are not using Visual Studio, Azure CLI 2.0 is the fallback. Run az login and go!

Service Principals

  • RunAs=App;AppId={AppId};TenantId=NotUsed;CertificateThumbprint={Thumbprint};CertificateStoreLocation={LocalMachine or CurrentUser}
  • RunAs=App;AppId={AppId};TenantId=NotUsed;CertificateSubjectName={Subject};CertificateStoreLocation={LocalMachine or CurrentUser}
  • RunAs=App;AppId={AppId};TenantId=NotUsed;AppKey={ClientSecret}
When used together with `KeyVaultClient`, the TenantId part of the connection string is not used, although `AzureServiceTokenProvider` throws an exception if it not provided. This is something that will [change in an upcoming version](https://github.com/Azure/azure-sdk-for-net/issues/4169) of `Microsoft.Azure.Services.AppAuthentication`.

Azure Managed Service Identity

Azure has a service called Managed Service Identity (MSI) which essentially provides service principals which are maintained by Azure. MSI is supported in App Service, Functions and Virtual Machines. See the official docs for more details.

  • RunAs=App;
This is fantastic! No more hassling about generating certificates or rotating secrets.

Automatic Mode

If no connection string is provided, AzureServiceTokenProvider will try to resolve tokens in the following order:

  1. MSI
  2. Visual Studio
  3. Azure CLI
If you use MSI in production and Visual Studio or Azure CLI for development, there is no need for any configuration. Yeah!