Hi,
We have a Active/Passive cluster set up with 7 instances of SQL 2005 SP4 Enterprise(64 bit).
Daily at 2 AM the SQL Cluster group is restarting on same node or failing over to the otehr node with errors starting with:
SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file in database. The OS file handle is 0x0000000000000B98. The offset of the latest long I/O is: 0x000012b0dfe000
/*there are many occurences of the above message on many instances and particularly msdb database*/
Event Id 4156 MSDTC - String message: ProcId = 0x2188 CSO: Maintain session; Received E_CM_SERVER_NOT_READY.
Event ID 19019
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 79; message = [Microsoft][SQL Native Client]TCP Provider: The semaphore timeout period has expired.
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 79; message = [Microsoft][SQL Native Client]Communication link failure
[sqsrvres] OnlineThread: QP is not online.
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure
Above errors are logged in the system logs and all the sql cluster groups are either restarting or failing over to other node every day from past one week around 2AM.
Backup jobs are scheduled to run at 2 AM on 4 instances same time every day.Could you please tell me if this is related to the backup jobs.We have different drives(for every instance and also different data/log/tempdb drives).
If this is not due to these backup jobs,could you please tell me how i can troubleshoot on this.I am suspecting a memory/io bottleneck but why is it failing now only which is running from from past few years.Please tell me which performance counters i can collect to resolve this issue.
We have a Active/Passive cluster set up with 7 instances of SQL 2005 SP4 Enterprise(64 bit).
Daily at 2 AM the SQL Cluster group is restarting on same node or failing over to the otehr node with errors starting with:
SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file in database. The OS file handle is 0x0000000000000B98. The offset of the latest long I/O is: 0x000012b0dfe000
/*there are many occurences of the above message on many instances and particularly msdb database*/
Event Id 4156 MSDTC - String message: ProcId = 0x2188 CSO: Maintain session; Received E_CM_SERVER_NOT_READY.
Event ID 19019
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 79; message = [Microsoft][SQL Native Client]TCP Provider: The semaphore timeout period has expired.
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 79; message = [Microsoft][SQL Native Client]Communication link failure
[sqsrvres] OnlineThread: QP is not online.
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure
Above errors are logged in the system logs and all the sql cluster groups are either restarting or failing over to other node every day from past one week around 2AM.
Backup jobs are scheduled to run at 2 AM on 4 instances same time every day.Could you please tell me if this is related to the backup jobs.We have different drives(for every instance and also different data/log/tempdb drives).
If this is not due to these backup jobs,could you please tell me how i can troubleshoot on this.I am suspecting a memory/io bottleneck but why is it failing now only which is running from from past few years.Please tell me which performance counters i can collect to resolve this issue.