I have SQL 2012 AlwaysOn set up on 2 FCIs (failover cluster instances). The purpose of AlwaysOn here is not for HA or DR. It's just to give me a real-time readable replica of 150+ production databases. The configuration is 1 primary FCI and 1 readable secondary FCI with asynch commit. The failover mode obviously is set to manual and we will never fail over using Always On under any circumstances. The read-write application connects to the primary FCI. An OLAP read-only application connects to the secondary FCI. It's been working without a problem for over 6 months.
A few days ago I was alerted to the fact that all of the AO database replicas on the primary were inaccessible. When I checked, they were all in a "restoring" state! When I checked AO on the primary it showed no groups and no replicas in either SSMS or the HADR DMVs.
On the secondary everything was still intact. The AO group was still there, it showed both the primary and secondary replicas as well as all of the databases replicas. I dont recall if the secondary DB replicas were in "synchronizing" or "not synchronized" state. I dont think thats important though. The important thing is that all traces of the AO group simply vanished from the primary replica.
To recover we had to delete the group from the secondary and rebuild it from scratch.
What could cause this?
In the SQL error log a few minutes before the corruption happened I see 3 messages referring to non-yielding dispatchers, then this- "The state of the local availability replica in availability group '<groupName>' has changed from 'PRIMARY_NORMAL' to 'RESOLVING_NORMAL'.", followed by 150+ messages saying that the AO connection with the secondary terminated. Its almost like it tried to do an AO failover automatically which is not configured.
Chuck