In enterprise virtualization infrastructures, protecting business continuity and data integrity is one of the top priorities of IT administrators. The connection between the ESXi hosts on which virtual machines (VMs) run and the storage systems (SAN/NAS Storage) hosting the data of these machines must be uninterrupted. However, hardware or software issues occurring in fiber channel (FC), iSCSI, or NFS networks can cause data storage units (LUN / Datastore) to become inaccessible by hosts.
When storage access issues occur in VMware vSphere environments, the ESXi host defines the situation with two different error scenarios: All Paths Down (APD) and Permanent Device Loss (PDL). These two situations are completely different from each other in terms of error characteristics and resolution methods. In this guide, we will discuss in detail the Permanent Device Loss (PDL) error, which can lead to data loss and virtual machine lockups, its causes, differences from APD, and step-by-step resolution methods.
What is Permanent Device Loss (PDL)?
Permanent Device Loss (PDL) is an error state in which an ESXi host definitely understands that it can no longer receive any response from the storage device (LUN) it is connected to, and that this situation is permanent.
In a PDL state, the storage unit (Storage) sends a specific response code to the ESXi host via the SCSI protocol. This response code (SCSI Sense Codes) informs the host that the device is no longer present, has been deleted, or the host's access permission on this LUN has been removed. When the ESXi host receives this message, it immediately stops searching for new connection paths for the relevant device and accepts that the device is permanently lost.
What are the Differences Between PDL and APD (All Paths Down)?
To make a correct diagnosis in storage issues, it is very important to know the differences between PDL and APD:
- Permanent Device Loss (PDL): It is certain that the storage device is permanently gone. The storage controller sends a "no device" message to the host. The ESXi host does not consume system resources by trying to reconnect unnecessarily. Virtual machines can be quickly included in failover (HA) processes.
- All Paths Down (APD): There is no definite information about the status of the storage device. Only all physical paths (cables, switches, etc.) between the host and the storage device are broken. The ESXi host continuously tries to reconnect, hoping that the device might return. This situation can cause the host to become unresponsive (host disconnect). For detailed information on this topic, you can review our VMware All Paths Down (APD) Error guide.
Primary Causes of VMware PDL Error
The most common scenarios that cause ESXi hosts to pull a storage unit into a PDL state are:
- Accidental Deletion of LUN by Storage: The storage administrator deleting or unmapping a LUN actively used on ESXi hosts from the storage interface.
- LUN Masking and Permission Changes: A change in access control lists (ACL) on the storage resulting in the removal of the ESXi host's access permission (mapping) to the relevant LUN.
- Storage Controller Failures: Failure of a control card on the storage and sending incorrect SCSI status codes to the host.
- Hardware RAID Losses: Complete collapse of the disk group (RAID) on the storage and physical disappearance of the LUN.
How to Diagnose PDL Error?
When a datastore becomes inaccessible, ESXi host logs and vSphere Client warnings must be examined to understand whether the issue is PDL.
1. vSphere Client Warnings
When the relevant ESXi host is selected on the vSphere Client, the following warnings are seen in the Events tab:
- Device <naa.id> has been permanently lost.
- Connectivity to storage device <naa.id> has been lost.
2. VMkernel Log Analysis
When the /var/log/vmkernel.log file is examined by connecting to the ESXi host via SSH, SCSI error codes confirming the PDL state are seen:
VMW_SATP_ALUA: satp_alua_issueCommandOnPath: Device is permanently inaccessible (SCSI_SENSE_KEY = 0x05, ASC = 0x25, ASCQ = 0x00)
- Sense Key 0x05 (Illegal Request) and ASC/ASCQ 0x25/0x00 (Logical Unit Not Supported) codes indicate that the storage device is telling the host "this LUN is no longer defined in me" and is a definitive PDL indicator.
Step-by-Step VMware PDL Error Resolution Methods
When you encounter a PDL error, you must apply the following steps in order to protect the health of virtual machines and hosts:
1. Check the Configuration on the Storage Side
If the LUN was accidentally unmapped or its permissions were changed:
- Log in to the storage management interface.
- Check whether the relevant LUN is correctly mapped to the ESXi host group.
- If permission was removed, map the LUN to the hosts again and run the Storage Rescan process on the ESXi hosts.
2. Verify vSphere HA (High Availability) Configuration
VMware vSphere features advanced capabilities to protect virtual machines automatically in case of a PDL state. The vSphere HA VM Component Protection (VMCP) feature must be activated:
- Go to vSphere Cluster settings.
- Click vSphere Availability -> Edit.
- Under Host Influence, set Datastore with PDL to Power off and restart VMs (Power off virtual machines and restart them on other hosts). Thanks to this, VMs on the host experiencing PDL will be automatically migrated to healthy hosts and powered on.
3. Safely Remove the Lost Device from the ESXi Host
If the LUN was deleted intentionally but still appears defined on the ESXi host, remove the device via CLI to prevent the host from becoming unstable:
- Connect to the ESXi host via SSH.
- Check the status of the disconnected device:
esxcli storage core device list -d <naa.id> - Completely remove the device from the host:
esxcli storage core device set --state=offline -d <naa.id> esxcli storage core device remove -d <naa.id>
Professional Solutions for Your Storage and Virtualization Infrastructure
Preventing critical storage errors such as PDL and APD in VMware vSphere environments requires a correct SAN/NAS architecture and redundant network design. For the installation of your enterprise storage systems, fiber switch configurations, and redundant path (Multipathing) settings, you can benefit from our NAS/SAN Storage Deployment and Configuration services.
To identify performance bottlenecks of your virtualization infrastructure, optimize your ESXi hosts, and correctly design high availability (HA) architectures, you can examine our VMware, Hyper-V and Proxmox Deployment Service solutions.
To professionally plan your business's information security, business continuity, and technology infrastructure roadmap, you can work with our expert team within the scope of our Business and Management Consulting services.
You can review our other technical guides that will strengthen your virtualization, storage, and information security infrastructure:
- For detailed audit records at the ESXi host level: VMware ESXi Audit Log and ISO 27001 Compliance
- For virtual infrastructure monitoring and performance tracking: ISO 27001 VMware Monitoring Methods
- For security of the vCenter management layer: vCenter Security and ISO 27001 Compliance
- For access authorization in the virtualization layer: ISO 27001 VMware Authorization Standards
- For ESXi host hardening steps: VMware ESXi Hardening Guide and ISO 27001
- For disk encryption requirements of virtual machines: VMware Datastore Encryption and ISO 27001
- For virtual isolation techniques at the network level: ISO 27001 VMware Network Isolation
- For logging requirements in virtual environments under KVKK: How to Configure VMware Logging for KVKK?
- For logging obligations under Law No. 5651: What is 5651 Logging, and for Whom is it Mandatory?
- For SIEM, Syslog, and 5651 architectures: SIEM, Syslog, and 5651 Correct Architecture Design
- For the foundation of information security standards: What is ISO 27001?
To optimize your VMware vSphere virtualization and storage infrastructure, and establish redundant and secure architectures against critical storage errors such as PDL/APD, you can contact us at any time.
Frequently Asked Questions
What happens to virtual machines during a PDL error?
When a PDL error occurs, disk access of virtual machines running on the relevant LUN is instantly cut off. This situation causes VM operating systems to switch to read-only mode, lock up, or crash completely (blue screen / kernel panic). If VMCP (VM Component Protection) is activated on vSphere HA, the system detects this situation and automatically restarts VMs on healthy hosts.
Is the PDL error a hardware failure or a software issue?
The PDL error can stem from both. Although it is often caused by software/configurational errors such as the storage administrator accidentally removing LUN permissions, it can also be triggered due to hardware failures such as storage controller failures or complete collapse of the disk group (RAID).
Should I reboot the host when I receive a PDL warning on the ESXi host?
No, you should not reboot immediately. In a PDL state, unlike the APD state, the ESXi host does not lock up and continues to operate stably. Instead of rebooting the host, you should first check the LUN mapping status on the storage side, and if the LUN was permanently deleted, you should safely remove the device from the host using CLI commands.
Conclusion
The Permanent Device Loss (PDL) error in VMware vSphere environments is a critical situation indicating that data storage units have become permanently inaccessible and requires rapid intervention. Unlike the APD error, in this scenario where the ESXi host definitely knows that the device is permanently lost, the active presence of vSphere HA VMCP features plays a vital role in protecting the business continuity of virtual machines. A correctly configured storage multipathing architecture, regular hardware checks, and professional network design will completely secure your virtual infrastructure against such critical storage outages.
