Wednesday, 31 July 2024

Azure Monitor Logs overview

 

Azure Monitor Logs overview

Azure Monitor Logs is a centralized software as a service (SaaS) platform for collecting, analyzing, and acting on telemetry data generated by Azure and non-Azure resources and applications.

You can collect logs, manage data models and costs, and consume different types of data in one Log Analytics workspace, the primary Azure Monitor Logs resource. This means you never have to move data or manage other storage, and you can retain different data types for as long or as little as you need.

This article provides an overview of how Azure Monitor Logs works and explains how it addresses the needs and skills of different personas in an organization.

Log Analytics workspace

Log Analytics workspace is a data store that holds tables into which you collect data.

To address the data storage and consumption needs of various personas who use a Log Analytics workspace, you can:

A screenshot of a Log Analytics workspace in the Azure portal.

You can also configure network isolation, replicate your workspace across regions, and design a workspace architecture based on your business needs.

Kusto Query Language (KQL) and Log Analytics

You retrieve data from a Log Analytics workspace using a Kusto Query Language (KQL) query, which is a read-only request to process data and return results. KQL is a powerful tool that can analyze millions of records quickly. Use KQL to explore your logs, transform and aggregate data, discover patterns, identify anomalies and outliers, and more.

Log Analytics is a tool in the Azure portal for running log queries and analyzing their results. Log Analytics Simple mode lets any user, regardless of their knowledge of KQL, retrieve data from one or more tables with one click. A set of controls lets you explore and analyze the retrieved data using the most popular Azure Monitor Logs functionality in an intuitive, spreadsheet-like experience.

Screenshot that shows Log Analytics Simple mode.

Users who are familiar with KQL can use Log Analytics KQL mode to edit and create queries, which they can then use in Azure Monitor features such as alerts and workbooks, or share with other users.

For a description of Log Analytics, see Overview of Log Analytics in Azure Monitor. For a walkthrough of using Log Analytics features to create a simple log query and analyze its results, see Log Analytics tutorial.

Built-in insights and custom dashboards, workbooks, and reports

Many of Azure Monitor's ready-to-use, curated Insights experiences store data in Azure Monitor Logs, and present this data in an intuitive way so you can monitor the performance and availability of your cloud and hybrid applications and their supporting components.

A screenshot that shows the built-in Container Insights monitoring experience in Azure Monitor.

You can also create your own visualizations and reports using workbooks, dashboards, and Power BI.

Table plans

You can use one Log Analytics workspace to store any type of log required for any purpose. For example:

  • High-volume, verbose data that requires cheap long-term storage for audit and compliance
  • App and resource data for troubleshooting by developers
  • Key event and performance data for scaling and alerting to ensure ongoing operational excellence and security
  • Aggregated long-term data trends for advanced analytics and machine learning

Table plans let you manage data costs based on how often you use the data in a table and the type of analysis you need the data for.

The diagram and table below compare the Analytics, Basic, and Auxiliary table plans. For information about interactive and long-term retention, see Manage data retention in a Log Analytics workspace. For information about how to select or modify a table plan, see Select a table plan.

Diagram that presents an overview of the capabilities provided by the Analytics, Basic, and Auxiliary table plans.

FeaturesAnalyticsBasicAuxiliary (Preview)
Best forHigh-value data used for continuous monitoring, real-time detection, and performance analytics.Medium-touch data needed for troubleshooting and incident response.Low-touch data, such as verbose logs, and data required for auditing and compliance.
Supported table typesAll table typesAzure tables that support Basic logs and DCR-based custom tablesDCR-based custom tables
Log queriesFull query capabilities.Full Kusto Query Language (KQL) on a single table, which you can extend with data from an Analytics table using lookup.Full KQL on a single table, which you can extend with data from an Analytics table using lookup.
Query performanceFastFastSlower
Good for auditing. Not optimized for real-time analysis.
Alerts
Insights
Dashboards✅ Cost per query for dashboard refreshes not included.Possible, but slow to refresh, cost per query for dashboard refreshes not included.
Data export
Microsoft Sentinel
Search jobs
Summary rules✅ KQL limited to a single table✅ KQL limited to a single table
Restore
Query price included
Ingestion costStandardReducedMinimal
Interactive retention30 days (90 days for Microsoft Sentinel and Application Insights).
Can be extended to up to two years at a prorated monthly long-term retention charge.
30 days30 days
Total retentionUp to 12 yearsUp to 12 yearsUp to 12 years*
*Public preview limitation: Auxiliary plan total retention is currently fixed at 365 days.


Data collection

To collect data from a resource to your Log Analytics workspace:

  1. Set up the relevant data collection tool based on the table below.
  2. Decide which data you need to collect from the resource.
  3. Use transformations to remove sensitive data, enrich data or perform calculations, and filter out data you don't need, to reduce costs.

This table lists the tools Azure Monitor provides for collecting data from various resource types

Resource typeData collection toolCollected data
AzureDiagnostic settingsAzure tenant - Microsoft Entra audit logs provide sign-in activity history and audit trail of changes made within a tenant.
Azure resources - Logs and performance counters.
Azure subscription - Service health records along with records on any configuration changes made to the resources in your Azure subscription.
ApplicationApplication insightsApplication performance monitoring data.
ContainerContainer insightsContainer performance data.
Virtual machineData collection rulesMonitoring data from the guest operating system of Azure and non-Azure virtual machines.
Non-Azure sourceLogs Ingestion APIFile-based logs and any data you collect from a monitored resource.


Working with Microsoft Sentinel and Microsoft Defender for Cloud

Microsoft Sentinel and Microsoft Defender for Cloud perform Security monitoring in Azure.

These services store their data in Azure Monitor Logs so that it can be analyzed with other log data collected by Azure Monitor.

No comments:

Post a Comment