DPM 2007 SP1 – Hyper-V – DPM encountered a retryable VSS error. (ID 30112 Details: Unknown error (0×800423f3) (0×800423F3))
A few days ago our Data Protection Manager (DPM) 2007 SP1 server started having issues with one specific Hyper-V Virtual Machine backup. The initial error reported in DPM was replica is inconsistent, with the full details:
Affected area: \Backup Using Child Partition Snapshot\VMSERVER1Occurred since: 2/19/2009 10:59:14 AMDescription: The replica of Microsoft Hyper-V \Backup Using Child Partition Snapshot\VMSERVER1 on VMSERVER1.domain.local is inconsistent with the protected data source. All protectionactivities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replicais consistent (ID 3106)DPM encountered a retryable VSS error. (ID 30112 Details: Unknown error (0×800423f3) (0×800423F3))Recommended action: Check the Application Event Log on HYPERVHOST.domain.local for the cause of the failure. Fix the cause and retry the operation.For more information on this error, go to http://go.microsoft.com/fwlink/?LinkId=132612.Synchronize with consistency check.Run a synchronization job with consistency check…Resolution: To dismiss the alert, click belowInactivate alert
I did some research but couldn’t find anything that really related to the issue I was having. There is a small bit of information on a Microsoft Technet article: Prerequisites and Known Issues with Hyper-V Protection Which mentions the error message I was receiving but also a second error which appears in the VMMS log on the Hyper-V host (in bold):
Hyper-V virtual machine backups (backups using child partition) fail with following error for multiple volume virtual machines: DPM encountered a retryable VSS error. (ID 30112 Details: Unknown error (0×800423f3) (0×800423F3)).
On the Hyper-V host in the event log: VMMS Event log contains following entry:
Failed to revert to VSS snapshot on one or more virtual hard disks of the virtual machine ‘%1′. (Virtual machine ID %2).
The recommended fix was to install KB959962 on the Hyper-V host, however this was already installed. After I double checked the VMMS log on the Hyper-V host I was unable to find a matching error in the VMMS log. I did however notice the following error:
Log Name: Microsoft-Windows-Hyper-V-VMMS-AdminSource: Microsoft-Windows-Hyper-V-VMMSDate: 19/02/2009 1:00:43 PMEvent ID: 10102Task Category: NoneLevel: ErrorKeywords:User: SYSTEMComputer: HYPERVHOST.domain.localDescription:Failed to create the backup of virtual machine ‘VMSERVER1′. (Virtual machine ID B8A0CE30-C40C-4FFD-891A-0E5A345B206E)
My next course of troubleshooting was to look directly on the Hyper-V Virtual Machine. At first I thought it may have been an issue with the Integration Services so I inserted the integration services disk and it alerted me that it needed to be updated. I let it run through and update the install. After it completed I restarted the Virtual Machine and tried running another synchronization job with consistency check. Again I got the same error.
Next I loaded up the event log on the Virtual Machine and I noticed multiple errors in the event log corresponding with similar times I was trying to run the synchronization job with consistency check:
Log Name: C:\tools\vmapplog.evtSource: VSSDate: 19/02/2009 11:15:27 AMEvent ID: 6013Task Category: NoneLevel: ErrorKeywords: ClassicUser: N/AComputer: VMSERVER1Description:The description for Event ID 6013 from source VSS cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.If the event originated on another computer, the display information had to be saved with the event.The following information was included with the event:ICommandText::Execute0×80040e14SQLSTATE: 42000, Native Error: 3013Error state: 1, Severity: 16Source: Microsoft OLE DB Provider for SQL ServerError message: BACKUP DATABASE is terminating abnormally.SQLSTATE: 42000, Native Error: 3271Error state: 1, Severity: 16Source: Microsoft OLE DB Provider for SQL ServerError message: A nonrecoverable I/O error occurred on file “{A484BA9E-9E14-43DB-8BE1-48C4622A8A23}6:” 995(error not found).the message resource is present but the message is not found in the string/message table
After seeing this I checked to see what service pack level the Microsoft SQL Server 2005 was at, I discovered that it had no service pack applied. So I proceeded to install Service Pack 3 and rebooted the Virtual Machine. Once this completed I ran another synchronization job with consistency check, this time I received a new error in DPM:
Affected area: \Backup Using Child Partition Snapshot\VMSERVER1Occurred since: 2/19/2009 1:15:58 PMDescription: The replica of Microsoft Hyper-V \Backup Using Child Partition Snapshot\VMSERVER1 on VMSERVER1.domain.local is inconsistent with the protected data source. All protectionactivities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replicais consistent (ID 3106)DPM is out of disk space for the replica. (ID 58 Details: There is not enough space on the disk (0×80070070))Recommended action: 1) Allocate more disk space for the replica. For more information, see “How to modify disk allocation” in DPM Help. 2) Rerun failed operation by clicking the link below.Synchronize with consistency check.Modify Disk Allocation …Run a synchronization job with consistency check…Resolution: To dismiss the alert, click belowInactivate alert
Woo! Progress. Next I increased the space for the replica in DPM. And ran another synchronization job with consistency check and it completed successfully!
I hope this post helps anyone who has a similar issue to me.
Bookmark/Search this post with:
