Hi All,
My SQL Server crashed a few times... and the error in the sql server error log viewer was the same - "...fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION..."
Since sql server saved a minDump file prior to crashing, I'm trying to read this file with WinDbg.exe to further understand where the problem is coming from....
I'm doing as guided on the web. Since my SQL Server machine is not connected to the web and I need to get the symbols from the microsoft symbol server than I didn't installed the WinDbg on my computer and loaded the mini-dump file onto it. But I'm running into a few issues.
Firstly when I open the mini-dump file I get this:
**** (start of past 1)
Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [F:\SQLDump0032.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Comment: 'Program fault handler'
Comment: '03/24/14 16:02:05 spid 111 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION reading address 0000000100000000 at 0x000000006FD98875
'
WARNING: Whitespace at end of path element
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (24 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon Mar 24 16:02:06.000 2014 (GMT+2)
System Uptime: 1 days 12:01:02.814
Process Uptime: 0 days 1:45:13.000
................................................................
.................................
Loading unloaded module list
..........................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(10e0.1f9c): Access violation - code c0000005 (first/second chance not available)
ntdll!NtWaitForSingleObject+0xa:
00000000`7751135a c3 ret
*** (end of paste 1)
Ok, so I checked things out on the web and ntdll!NtWaitForSingleObject seems to be a lock issue (which is quite surprising - why would sql server crash from that especially if we have a deadlock mechanism).
Anyways, I wanted to investigate further and did .excr:
**** (start of paste 2)
0:000> .ecxr
rax=0000000004445ff0 rbx=0000000000000002 rcx=0000000021976e90
rdx=00000000823661a0 rsi=0000000000000000 rdi=00000000219774a8
rip=000000006fd98875 rsp=0000000021976e18 rbp=0000000021977168
r8=0000000004446b40 r9=0000000021977d40 r10=000000006fd80000
r11=0000000000000000 r12=0000000021977d40 r13=0000000100000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
msvcr80!output_l+0x2b5:
00000000`6fd98875 410fb67d00 movzx edi,byte ptr [r13] ds:00000001`00000000=??
**** (end of paste 2)
I really don't understand much, so I realized that though i've defined the "symbol file path", I probably still need to run ".reload /f" but there are many errors especially one of sqlservr.exe (which is the most important) - why is this happening is it because the mini-dump is read from a machine different there where the dump happened (or maybe because this machine also doesn't have sql server installed on it???), anyway that's what I got:
*** (start of paste 3)
0:000> .reload /f
.Unable to load image C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe, Win32 error 0n2
*** WARNING: Unable to verify timestamp for sqlservr.exe
........Unable to load image C:\Windows\winsxs\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\msvcr80.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for msvcr80.dll
.Unable to load image C:\Windows\winsxs\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\msvcp80.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for msvcp80.dll
.Unable to load image C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlos.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for sqlos.dll
..............ECT........
*** (end of paste 3)
Moving on I also tried to run the command "!locks" and it also failed:
*** (start of paste 4)
0:000> !locks
NTSDEXTS: Unable to resolve ntdll!RTL_CRITICAL_SECTION_DEBUG type
NTSDEXTS: Please check your symbols
*** (end of paste 4)
So, basically I need some help....:
1. How do I go about finding out why sql server keeps crashing (every few hours)?
2. How do I read the mini-dump and load symbols correctly to WinDbg.exe (do I need to run on the same machine as when the sql server that crashed is - or do i need to be a machine that has sql server on it?
3. Also, how do I accurately understand from WinDbg.exe what is exactly going on and where does the crash originate...
Thanks in advance,
Dror