Azure VM Agent and Extension Logs¶

This is a collection of links and file locations used to troubleshoot various Azure VM extensions or agents. Mainly done so I don't have to google it everytime

Azure virtual machine extensions and features

Azure Windows VM Agent / waagent¶

WindowsLinux

Quote

The Microsoft Azure Windows VM Agent is a secure, lightweight process that manages virtual machine (VM) interaction with the Azure fabric controller. The Azure Windows VM Agent has a primary role in enabling and executing Azure virtual machine extensions. VM extensions enable post-deployment configuration of VMs, such as installing and configuring software. VM extensions also enable recovery features such as resetting the administrative password of a VM. Without the Azure Windows VM Agent, you can't run VM extensions.

Quote

The Microsoft Azure Linux VM Agent (waagent) manages Linux and FreeBSD provisioning, along with virtual machine (VM) interaction with the Azure fabric controller. In addition to the Linux agent providing provisioning functionality, Azure provides the option of using cloud-init for some Linux operating systems.

https://github.com/Azure/WindowsVMAgent

https://github.com/Azure/WALinuxAgent

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/agent-windows

https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows-azure-guest-agent

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/troubleshoot

WindowsLinux

Name	Location
Name	`WindowsVMAgent` (1)
Log file	`C:\WindowsAzure\Logs\WaAppAgent.log`
Installer logs	`C:\WindowsAzure\Logs\TransparentInstaller.log`
Extension logs	`C:\WindowsAzure\Logs\Plugins\`
Extension config	`C:\Packages\Plugins\`

Note: The Windows process is called WindowsAzureGuestAgent.exe

Name	Location
Name	`WALinuxAgent`
Config file	`/etc/waagent.conf`
Log file	`/var/log/waagent.log`
Extension logs	`/var/log/azure/<extensionName>`
Extension binaries	`/var/lib/waagent/<extensionName>`
Verify agent is running	`systemctl status walinuxagent`
journald logs	`journalctl -u walinuxagent`

Azure Arc Connected Machine Agent¶

https://learn.microsoft.com/en-us/azure/azure-arc/servers/agent-overview

https://learn.microsoft.com/en-us/azure/azure-arc/servers/troubleshoot-agent-onboard

https://learn.microsoft.com/en-us/azure/azure-arc/servers/troubleshoot-vm-extensions#general-troubleshooting

https://learn.microsoft.com/en-us/azure/azure-arc/servers/azcmagent-check

https://learn.microsoft.com/en-us/azure/azure-arc/servers/agent-overview#agent-resources

Arc enabled servers support the extensions listed in the links for Windows and Linux.

For example, the Qualys extension used in the article on how to deploy the Qualys extension with Azure Policy is not available for arc enabled servers.

WindowsLinux

Name	Location
Connected Machine Agent log	`%ProgramData%\AzureConnectedMachineAgent\Log\azcmagent.log`
Hybrid Instance Metadata Service (himds) log (2)	`%ProgramData%\AzureConnectedMachineAgent\Log\himds.log`
Guest agent log	`%SystemDrive%\ProgramData\GuestConfig\ext_mgr_logs`
Specific extension logs	`%SystemDrive%\ProgramData\GuestConfig\extension_logs\<Extension>`
Download path for VM extensions	`%SystemDrive%\%ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\downloads`
Extension install dir	`%SystemDrive%\Packages\Plugins\<extension>` (1)

Note: Same path as the WindowsAzureGuestAgent installs extensions to
Metadata information about a connected machine is collected after the Connected Machine agent registers with Azure Arc-enabled servers

Collect all logs and store in a ZIP file:

azcmagent logs --full --output "C:\temp\azcmagent-logs.zip"

Name	Location
Connected Machine Agent log	`/var/opt/azcmagent/log/azcmagent.log`
Hybrid Instance Metadata Service (himds) log (2)	`/var/opt/azcmagent/log/himds.log`
Guest agent log	`/var/lib/GuestConfig/ext_mgr_logs`
Specific extension logs	`/var/lib/GuestConfig/extension_logs/`
Download path for VM extensions	`/opt/GC_Ext/downloads`
Extension install dir	`/var/lib/waagent/<extension>` (1)

Note: Same path as the WALinuxAgent installs extensions to
Metadata information about a connected machine is collected after the Connected Machine agent registers with Azure Arc-enabled servers

Collect all logs and store in a ZIP file

azcmagent logs --full --output "/tmp/azcmagent-logs.zip"

Agent connectivity check:

azcmagent check --location "westeurope" -v

Azure Monitor Agent¶

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/use-azure-monitor-agent-troubleshooter

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-data-collection-endpoint

Info

A system-assigned or user-assigned managed identity is necessary for the agent to work

WindowsLinux

Name	Location
Extension name	`AzureMonitorWindowsAgent`
Runtime logs	`C:\Resources\Azure Monitor Agent\`
Binary store	`C:\Program Files\Azure Monitor Agent\`
Run installation with logging enabled	`Msiexec /I AzureMonitorAgentClientSetup.msi /L*V <logfilename>`

Run Windows Troubleshooter:

cd "C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\{version}\Troubleshooter"
AgentTroubleshooter --ama

Name	Location
Extension name	`AzureMonitorLinuxAgent`
Core agent logs	`/var/opt/microsoft/azuremonitoragent/log/mdsd.*`
Download location for DCRs	`/etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/`
Configuration store	`/etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/`
Verify agent is running	`systemctl status azuremonitoragent`

Run Linux Troubleshooter:

cd /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-{version}/ama_tst
sudo sh ama_troubleshooter.sh -A

# create zip file with logs (you'll be asked for a file location to create the zip file):
sudo sh ama_troubleshooter.sh -A L

Guest Configuration Extension¶

My first steps in diving into Azure Guestconfiguration (Azure Automanage Machine Configuration) are written down in part 1 and part 2.

https://learn.microsoft.com/en-us/azure/governance/machine-configuration/overview

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/guest-configuration

https://learn.microsoft.com/en-us/azure/governance/machine-configuration/agent-release-notes

https://github.com/Azure/azure-policy/tree/master/built-in-policies/policySetDefinitions/Guest%20Configuration

https://github.com/Azure/azure-policy/tree/master/samples/GuestConfiguration/package-samples/resource-modules

https://github.com/azure/nxtools#getting-started

Info

A system-assigned managed identity is necessary for the agent to work

WindowsLinux

Name	Location
Extension name	`AzurePolicyforWindows`
Client log files Azure VM	`C:\ProgramData\GuestConfig\gc_agent_logs\gc_agent.log`
Client log files Arc	`C:\ProgramData\GuestConfig\arc_policy_logs\gc_agent.log`
Agent files (1)	`C:\ProgramData\guestconfig\configuration\`

The machine configuration agent downloads content packages to a machine and extracts the contents

Name	Location
Extension name	`AzurePolicyforLinux`
Client log files Azure VM	`/var/lib/GuestConfig/gc_agent_logs/gc_agent.log`
Client log files Arc	`/var/lib/GuestConfig/arc_policy_logs/gc_agent.log`
Agent files (1)	`/var/lib/GuestConfig/Configuration/`

The machine configuration agent downloads content packages to a machine and extracts the contents

Custom Script Extension¶

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux

https://github.com/Azure/custom-script-extension-linux

WindowsLinux

Name	Location
Extension name	`CustomScriptExtension`
Extension output filepath	`C:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension\`
Extension file download path	`C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.*\Downloads\<n>` (1)

In the preceding path, <n> is a decimal integer that might change between executions of the extension. The 1.* value matches the actual, current typeHandlerVersion value of the extension. For example, the actual directory could be C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Downloads\2. Read here

Name	Location
Extension name	`customScript`
waagent logs containing `customScript`	`sudo cat /var/log/waagent.log \| grep "Microsoft.Azure.Extensions.customScript`
Extension logs	`/var/log/azure/custom-script/handler.log`
Extension file download path	`/var/lib/waagent/custom-script/download/0/`

Dependency Agent¶

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/agent-dependency-windows

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/agent-dependency-linux

WindowsLinux

Name	Location
Extension name	`DependencyAgentWindows`
Extension logs	`C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Monitoring.DependencyAgent\`

Name	Location
Extension name	`DependencyAgentLinux`
Extensions logs	`/var/opt/microsoft/dependency-agent/log/install.log`

Network Watcher Extension¶

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/network-watcher-linux

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/network-watcher-windows

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/network-watcher-update

WindowsLinux

Name	Location
Extension name	`NetworkWatcherAgentWindows`
Extension logs	`C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentWindows\`

Name	Location
Extension name	`NetworkWatcherAgentLinux`
Extension logs	`/var/log/Microsoft/Azure/NetworkWatcherAgent/Logs`

Qualys VM extension¶

Please also see the article on how to deploy the Qualys extension with Azure Policy

WindowsLinux

Name	Location
Extension name	`QualysAgent`
Agent logs	`C:\WindowsAzure\Logs\Qualys.QualysAgent\3.1.3.34\Asclog.txt`
Agent installer	`C:\Packages\Plugins\Qualys.QualysAgent\3.1.3.34\`

Name	Location
Extension name	`QualysAgentLinux`
Agent logs	`/var/log/qualys/qualys-cloud-agent.log`
Extension logs	`/var/log/azure/Qualys.QualysAgentLinux/lxagent.log`
Install directory	`/usr/local/qualys/cloud-agent/bin`
Install scripts	`/var/lib/waagent/Qualys.QualysAgentLinux-1.6.1.4/bin/avme_install.sh`
Config files	`/etc/qualys/cloud-agent/`

Azure VM Agent and Extension Logs¶

Azure Windows VM Agent / waagent¶

Azure Arc Connected Machine Agent¶

Azure Monitor Agent¶

Guest Configuration Extension¶

Custom Script Extension¶

Dependency Agent¶

Network Watcher Extension¶

Qualys VM extension¶

Comments