Background

Many teams that host applications in Azure App Service setup the availability monitoring for their apps using Application Insights web tests, which can be basic URL ping checks as well as more advanced multi-step and performance ones. A common approach, when creating such tests, is also to configure alerts to notify or trigger an action on some criteria, i.e., when the tests fail.

As a result, people responsible for keeping the application alive are aware that something goes wrong with the application and can take remediation actions.

Also, it is a common practice to put an application or service in operation into the maintenance mode in a monitoring solution before applying any changes to it or updating its components, so as not to be wakened up at 2 am by false-positive alerts on your “pager.”

Problem

Regarding the Azure App Service applications in my example, such misleading notifications from the web test were triggered during the regular continuous deployments from Azure DevOps pipelines. This was not good, as each time the Ops have to verify with the Devs whether the issue was not fake.

Unfortunately, there is no such handy built-in maintenance mode for monitored items in Application Insights as, for example, in System Center Operations Manager, where you can suspend all alerting for a specific configuration item or a group of items for some period and not to worry about putting them back on monitoring later.

To make it worse, there are even no PowerShell cmdlets yet to work with the web tests specifically in Az 3.0.0:

Solution

Currently, one of the options to interact with ‘Microsoft.Insights/webtests’ resources is to use generic ‘Get-AzResource’ and ‘Set-AzResource’ PowerShell cmdlets. They might be not so convenient as specialized cmdlets, but they still do the job as you can manipulate resource properties.

For example, a script to disable all web tests in a specific resource group might look like the following:

To enable them back, you can modify the ‘Enabled’ property to ‘True’ and rerun the script.

Armed with these scripts, you can emulate the maintenance mode for your application in the deployment pipeline:

You can use the Azure PowerShell tasks to disable the tests before deployment and enable them back after it.

Also, don’t forget to configure the run conditions for the enabling task, so you have your monitoring turned on even if a previous task, e.g., a deployment, failed:

A sample YAML snippet for this:

P.S. I didn’t want to fall back to ARM REST APIs in my solution as their format usually makes the script logic hard to understand, and IMHO it is a complete nightmare for future solution maintenance.

How do you monitor your apps in production and put them in maintenance mode? Share your experience in the comments!