Restore operation failing in VSS return sVSS_E_OBJECT_NOT_FOUND

I am trying backup and restore of a MS SQL server through VSS. My backup operation is running fine and I store the backup metadata in XML file.
During restore operation I am providing the same XML I created during backup operation, what is happening is that restore is failing intermittently in PreRestore Stage returning VSS_E_OBJECT_NOT_FOUND. I have tried following things

1. Checked that if my VSS service is running correctly at the time of restore.
2. Checked Microsoft Shadow Copy provider is running during restore.
3. Checked if network services are running fine during restore.
4. Checked on different forum what this error could be related to but most of the suggestion are like trying reboot the system.

Following is the line where restore operation failed:

CHECK_COM(m_pVssObject->SetAdditionalRestores(myWriterId, VSS_CT_FILEGROUP, logicalPath, compName, true));

and this is the response it is returning.