Hi There,
I have a 3 node AlwaysOn Availability Group cluster. Each node a has vote for quoum and there is no witness.
Node 1 and 2 are in the same data center while node 3 is in a remote data center connected by a high speed layer 2 link. All nodes are on the same subset. Node 1 is a primary for one AG1 and node 2 is a primary for AG2 while node 3 is only a secondary for both
AGs.
The other night node 3 crashed (the hypervisor went down) which seems to have caused nodes 1 and 2 both to fail at exactly the same time a few minutes later, which seems odd as the whole point of the cluster is to avoid this scenario.
In the system event log I can see the below error on nodes 1 and 2 which I believe is the result a failed health check.
Looking in the failoverclustering diagnostic log on nodes 1 and 2 I see the this error just a second later and then the cluster service is terminated on both. I've searched the internet for references to "GumLockIsStuck" but come back with
nothing but I believe this is related to updating the cluster databases with the cluster status on all nodes.
Because the above error was referring to no reporting for 600 seconds I went back 10 minutes in the same log and could a whole bunch of errors like this one stating nodes 1 and 2 were unable receive an ACK from node 3 which had of course crashed (around this time).
So what looks like has happened is node 3 has gone down and nodes 1 and 2 can't update the status of the cluster and because of that they have then stopped the cluster service on themselves. This doesn't seem right at all so can anyone explain this behavior?
Cheers
C