Merge Variables in a Azure DevOps Pipeline

Azure DevOps Variable Groups are an effective and straightforward way to manage deployment-related settings. However, managing multi-stage pipelines, multiple variable groups, and templates can quickly become complex. This post shows a little trick we use to combine multiple variables into a single source of truth. That enables us to manage default settings in a centralized way, but still keep flexibility to manage app-specific settings. This is especially useful for managing Azure App Service or Azure Function App Settings.

What is the problem we are trying to solve?

Let’s assume we have created a release.yml that we want to use as a template for multiple deployment pipelines. To complicate matters further, we are working with a multi-stage pipeline that deploys to both development (dev) and production (prod) environments. For this example, we deploy an Azure Function and want to set several defaults in our release.yml template, as we aim to centralize the management of our default settings.

parameters:
- name: BuildConfiguration
  type: string
  default: "Release"
  values: 
    - Release
    - Debug

- name: FunctionName 
  type: string

variables:
  - name: BuildConfiguration 
    value: ${{ parameters.BuildConfiguration }}
  - name: FunctionName
    value: ${{ parameters.FunctionName }}
  - name: DevServiceConnection
    value: "azure-dev"
  - name: PrdServiceConnection
    value: "azure-prd"
  ...snip...

- stage: "Release_DEV"
  dependsOn: 
    - Build_Function
  condition: succeeded()
  displayName: "Release DEV"
  pool: 
    name: Default
  variables:
  - group: "DEV"
  # the variables are read from the variable group "DEV"
  - name: DefaultFunctionSettings
    value: >
      -AppConfigurationConnection "$(AppConfiguration_ManagedIdentity_ConnectionString)" 
      ...snip...
  jobs:
  - deployment: Deploy
    displayName: Deploy to DEV Environment
    environment: dev_environment
    strategy:
      runOnce:
        deploy:
          steps:
          - template: steps-app-deploy.yml
            parameters:
              ServiceConnectionName: $(DevServiceConnection)
- stage: "Release_PRD"
  dependsOn: 
    - Build_Function
  condition: succeeded()
  displayName: "Release PRD"
  pool: 
    name: Default
  variables:
  - group: "PRD"
  # the variables are read from the variable group "PRD"
  - name: DefaultFunctionSettings
    value: >
      -AppConfigurationConnection "$(AppConfiguration_ManagedIdentity_ConnectionString)" 
      ...snip...
  jobs:
  - deployment: Deploy
    displayName: Deploy to PRD Environment
    environment: prd_environment
    strategy:
      runOnce:
        deploy:
          steps:
          - template: steps-app-deploy.yml
            parameters:
              ServiceConnectionName: $(PrdServiceConnection)

To consume this pipeline template we include it in our release pipeline within our application repository:

trigger:
  batch: true
  branches:
    include:
      - master

parameters:
- name: BuildConfiguration
  displayName: Build Config
  type: string
  default: Release
  values: 
    - Release
    - Debug

name: $(Date:yyyyMMdd).$(BuildID)

resources:
  repositories:
    - repository: pipelines
      type: git
      name: WareTec.Pipelines

extends:
  template: Pipelines/Functions/stages-function-release.yml@pipelines
  parameters: 
    BuildConfiguration: ${{ parameters.BuildConfiguration }}
    FunctionName: "myApp"

This works fine, but what if we want to add additional settings per app? We cannot add another variables block to our pipeline, because we defined it already in our release.yml. We could move the variables block from the template to the pipeline definition in the repository, but that means we cannot manage our service connections or other important variables in a centralized way.

Merging Variables

Our solution involves adding an optional parameter to the template pipeline to define additional variables, enabling the merging of all configuration sources into a single source of truth.

parameters:
- name: BuildConfiguration
  type: string
  default: "Release"
  values: 
    - Release
    - Debug

- name: FunctionName 
  type: string

- name: AdditionalFunctionSettings
  type: string
  default: ""

variables:
  - name: BuildConfiguration 
    value: ${{ parameters.BuildConfiguration }}
  - name: FunctionName
    value: ${{ parameters.FunctionName }}

- stage: "Release_DEV"
  dependsOn: 
    - Build_Function
  condition: succeeded()
  displayName: "Release DEV"
  pool: 
    name: Default
  variables:
  - group: "DEV"
  # the variables are read from the variable group "DEV"
  - name: DefaultFunctionSettings
    value: >
      -AppConfigurationConnection "$(AppConfiguration_ManagedIdentity_ConnectionString)" 
      ...snip...
  - ${{ if parameters.AdditionalFunctionSettings }}:
    - name: MergedAppSettings
      value: $[ '${{ format('{1} {0}', parameters.AdditionalFunctionSettings, variables.DefaultFunctionSettings) }}' ]
  - ${{ else }}:
    - name: MergedAppSettings
      value: $[ '${{ variables.DefaultFunctionSettings }}' ]
  jobs:
  - deployment: Deploy
    displayName: Deploy to DEV Environment
    environment: dev_environment
    strategy:
      runOnce:
        deploy:
          steps:
          - template: steps-app-deploy.yml # Access the MergedAppSettings using $(MergedAppSettings) inside steps-app-deploy.yml
            parameters:
              ServiceConnectionName: $(DevServiceConnection)

Now that we’ve added the AdditionalFunctionSettings parameter, we can use it in our pipeline to define some app settings we want to include. These settings can also include variables, which will be replaced by the variables defined within the scope of the pipeline stage.

trigger:
  batch: true
  branches:
    include:
      - master

parameters:
- name: BuildConfiguration
  displayName: Build Config
  type: string
  default: Release
  values: 
    - Release
    - Debug

name: $(Date:yyyyMMdd).$(BuildID)

resources:
  repositories:
    - repository: pipelines
      type: git
      name: WareTec.Pipelines

extends:
  template: Pipelines/Functions/stages-function-release.yml@pipelines
  parameters: 
    BuildConfiguration: ${{ parameters.BuildConfiguration }}
    FunctionName: "myapp"
    AdditionalFunctionSettings: >
      -ServiceBusConnection "$(Wtc-Connections-ServiceBus)"

What happend?

Azure DevOps provides two distinct syntaxes for variable expressions, each with specific use cases and scopes:

• Runtime expressions ($[]): These are evaluated during runtime, meaning they are processed as the pipeline runs. They allow you to dynamically compute values based on runtime context.
• Template expressions (${{ }}): These are evaluated at compile time when the pipeline YAML is parsed and converted into an execution plan. They allow for conditional logic and template expansion but cannot access variables.

So after compile time our MergedAppSettings variable would look like this:

$[ 
  format('{1} {0}', '-AppConfigurationConnection "$(AppConfiguration_ManagedIdentity_ConnectionString)"', '-ServiceBusConnection "$(Wtc-Connections-ServiceBus)"') 
]

At runtime, the variables are replaced with their actual values, and the format() expression is executed. Consequently, the MergedAppSettings variable will contain the desired values.