Azure Monitor 101
Recently we have been involved in a project for our Customer, who owns web-based line of business application, which consists of four main parts:
- Web app (front-end) deployed on Azure App Service
- Database on Azure SQL database
- Reporting implemented in Azure Analysis Services
- Data exchange via Azure Data Factories
All vital elements of the business application exist in Azure framework and this situation leads us to the main project role of our DevOps team – configure monitoring environment for the reporting application based on native Azure Management services.
The monitoring solution should include performance metrics and logs collection, Azure portal dashboards and alert rules. Looks like quite ordinary goal, but the question is which Azure Management service is better to use? The answer is: Azure Monitor.
So, lets overview the Azure Monitor service. During subsequent paragraphs, we will describe and demonstrate Azure Monitor value on Azure SQL database example, as it came from the project in real world. If you run different type of resources via Azure, do not worry, discussed concepts are applicable for most resources managed by Azure Monitor.
Azure Monitor overview
What is Azure Monitor?
First, we need to know about Azure Monitor. It is a Microsoft Azure built-in platform monitoring service and it collects performance metrics, activity logs and diagnostics logs from Azure resources, processed in order to achieve the following goals:
Performance metrics visualization:
- Logs analysis
- Alerts notification
- Routing to another subsystem
- Automated actions
The overall monitoring process, based on Azure Monitor service, is shown on the image below.
Azure Monitor is a core monitoring service for Azure resources and it is the run-up point in monitoring, diagnostics and troubleshooting of Azure resources. For simplicity, we can compare it with performance metrics and event logs on Windows computer. Out of the box Windows gives many possibilities to find bottlenecks. When enterprise applications installed on Windows Server, usually they supplement app-specific performance counters and event logs into Windows. Therefore, generic and fast monitoring is always on board!
Similar situation with Azure Monitor – this service is always on hand and it is a best friend of a Cloud Engineer. It is a good thing to remember about it. For example, when a company begins to use a new Azure resource for their application, it can investigate if the monitoring of this resource supported by Azure Monitor. Assume that the company moved from on-premises Microsoft SQL Server to Azure SQL Database. Do they need operational monitoring? Indeed, from the beginning! They need to control facets such as a database size, IO or CPU utilization and fire alert in case of something bad. Currently Azure Monitor collects 15 performance metrics and 13 logs categories for Azure SQL Database out of the box and even more for other resources!
Illustrated example is on the image below.
It is worth to mention that metrics come to Azure Monitor storage very fast - time varies from five to *** drum roll *** 1 minute! This means that Azure Monitor is a real operational service and cannot be underestimated!
Resources managed by Azure Monitor
Not all resources send metrics and logs to Azure Monitor. That is why before starting the design of monitoring solution for your Azure application, be sure whether Azure Monitor currently supports the application resources. Azure Monitor is a rapidly developing service and it is possible that today's unmonitored resource will be monitored tomorrow. Common Azure resources are virtual machines, storage accounts, SQL databases, Web applications and Analysis services.
Azure Monitor consumption methods
Maybe someone wonder: "How do I start using the Azure Monitor?"
Yes, this is a right question!
Let us split usage methods into three categories as in the middle section of the image below.
Azure Portal is the easiest and the best way to start working with Azure Monitor. Azure Monitor elements are available on the portal in few places: in the context of a monitored resource, in the context of a resource group and in the dedicated pane. We will focus on the last one.
When you work with Azure Monitor on its dedicated pane, the main page of the service contains a subset of the centralized monitoring data available in one place. To view this main page, open Azure Portal, navigate to "All services" in the left upper corner, choose it and type the word "Monitor". Then you need to click the "Monitor" link. Screenshot below shows the search result.
Alternatively, Azure Monitor icon can be found in the left main blade on the Azure Portal if it is marked as favorite. On the screenshot below, Azure Monitor icon is highlighted in green.
There are inbuilt blades related to Azure Monitor. Let’s quickly describe few of them.
First blade is "Overview". It is a good place to start.
Next group of blades is “SHARED SERVICES”. Each blade serves different purposes:
- “Alerts” - manage alert rules based on Azure Monitor collected data and threshold values
- “Metrics” - use Metric Explorer and visualize collected performance counters for chosen resource across Azure Subscription(s)
- “Log Analytics” - easily open query page for particular Log Analytics workspace
- “Activity log” - investigate operations/actions across Azure Subscription(s)
On the example below you can see how to use Metric Explorer (with the new metric type) in order to investigate Azure SQL Database. For this example, we took the average values for the last 30 minutes with a metric name “Total database size”.
Azure Monitor shipped with Azure Monitor REST API, PowerShell cmdlets, Azure CLI support and .Net/Python/Node.js SDK. Programmatic access is great for custom monitoring, integration or scenarios automation.
For example, organization owns on-premises monitoring software and needs to get performance counter values for particular applications hosted in Azure. In this case, basing on a schedule, custom script can retrieve metrics values per resource directly from REST API or through SDK, format gathered results and put to their monitoring system.
Another example, company uses incident management system and track every action on the particular administrative level (like resource deletion).Such operations can be retrieved and stored in the incident management system via scheduled data requests.
Simple way to start using programmatic approach is PowerShell (Azure Resource Manager cmdlets must be installed or PowerShell can be executed without modules installation via Azure Cloud Shell). Execution of “Get-AzureRmMetric” cmdlet to get the metric values is on the image below. In this case, through PowerShell retrieved “Total storage size” metric values in bytes for last 30 minutes with time grain of 5 minutes and ‘average’ aggregation type (“storage” is the short name for metric). Azure SQL Database resource id sits in the $id variable.
Final out of box method of Azure Monitor consumption is data streaming. Data collected by Azure Monitor can be routed/streamed to different Azure services (Storage Account, Event Hub, and Log Analytics).
If there is a need of custom monitoring scenarios, not typical diagnostic data collection and/or correlation between different data sources, Azure Monitor can interact mutually with deep monitoring services like Azure Application Insights or Azure Log Analytics.
For example, on the mentioned project above, Azure SQL Database configured to send Azure Monitor metrics and diagnostic logs to Log Analytics workspace for deeper investigations and operations not available in the Azure Monitor.
Another example, to archive monitoring data longer than 90 days. Storage account is simple solution for this goal. In some cases, streaming to storage account, gives opportunity to read Azure Monitor data indirectly. Azure management services cooperation example for complex solutions showed on the picture below.
Many Microsoft partners already produced plugins or integrations with Azure Monitor to make integration process painless. Such integrations solutions rely on Azure Monitor REST API, streaming or Azure Monitor SDK. Therefore, the choice of user to use developed partners solutions or create own integrations. Among partner’s integrations are Graphana, Splunk, Dynatrace, AppDynamics, Moogsoft, Pagerduty and others. They provide different functionality in their integrations solutions.
Figuratively speaking, Azure Monitor is a first gear in gearbox! Monitoring of Azure resources starts here! If you need more information about Azure Monitor or your company maintains Azure based enterprise application and interested in well-designed monitoring solution – You Are Welcome!