Server crash - won't start (InnoDB error?)

Hi all,

Earlier today it seems there was a power loss at one of the
co-location facilities I use for my servers, all 4 servers i had there
restarted unexpectedly this morning, all the same minut. I'm still
waiting for an explanation from the colo host.

The problem is that the mysql server on one of the servers won't start
again. This server has a big database, with som 20 billion InnoDB
records and about 15 million in myisam tables.
The server runs MySQL 4.1.??, Linux Fedora core 4 on a dual Xeon
server with 4G ram.

When I try to start MySQL, I get this:

************************************************** *****************

070309 10:10:32 mysqld started
070309 10:10:32 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the
doublewrite
InnoDB: buffer...
070309 10:10:32 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 145 3929780998.
InnoDB: Doing recovery: scanned up to log sequence number 145
3931378469
070309 10:10:33 InnoDB: Starting an apply batch of log records to the
database...
InnoDB: Progress in percents: 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 97 98 99 mysqld got signal 11;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is
definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=33550336
max_used_connections=0
max_connections=512
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_connections = 4192252 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find
out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x483466a8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81356bf
0x47f420
0x4f40fc59
0x8348a4c
0x834b141
0x832f3a3
0x836af8d
0x8288364
0xa4fbd4
0x5684fe
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Us...ack_trace.html
and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please
do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
070309 10:10:33 mysqld ended

************************************************** *********

When resolving the stack dump I get this:

resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack
0x81356bf tab_gbk_uni0 + 33407
0x47f420 (?)
0x4f413c59 _end + 1193789273
0x8348a4c _end + 1774924
0x834b141 _end + 1784897
0x832f3a3 _end + 1670819
0x836af8d _end + 1915533
0x8288364 _end + 986724
0xa4fbd4 (?)
0x5684fe (?)

************************************************** *********

Unfortunately this doesn't tell me much. It seems to be an InnoDB
problem, since MySQL crashes when reaching 99% in the InooDB check...
But what can i do to solve it?

Any help is appreciated


/GoL

On Fri, 09 Mar 2007 14:11:40 +0100, GoL
<usenet@^^.wwwspace.net._nospam_^^> wrote:

>Hi all,
>
>Earlier today it seems there was a power loss at one of the
>co-location facilities I use for my servers, all 4 servers i had there
>restarted unexpectedly this morning, all the same minut. I'm still
>waiting for an explanation from the colo host.
>
>The problem is that the mysql server on one of the servers won't start
>again. This server has a big database, with som 20 billion InnoDB
>records and about 15 million in myisam tables.

correction: 15 million in myisam RECORDS, nothing else.

>The server runs MySQL 4.1.??, Linux Fedora core 4 on a dual Xeon
>server with 4G ram.
>
>When I try to start MySQL, I get this:
>
>************************************************* ******************
>
>070309 10:10:32 mysqld started
>070309 10:10:32 InnoDB: Database was not shut down normally!
>InnoDB: Starting crash recovery.
>InnoDB: Reading tablespace information from the .ibd files...
>InnoDB: Restoring possible half-written data pages from the
>doublewrite
>InnoDB: buffer...
>070309 10:10:32 InnoDB: Starting log scan based on checkpoint at
>InnoDB: log sequence number 145 3929780998.
>InnoDB: Doing recovery: scanned up to log sequence number 145
>3931378469
>070309 10:10:33 InnoDB: Starting an apply batch of log records to the
>database...
>InnoDB: Progress in percents: 7 8 9 10 11 12 13 14 15 16 17 18 19 20
>21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
>44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
>67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
>90 91 92 93 94 95 96 97 98 99 mysqld got signal 11;
>This could be because you hit a bug. It is also possible that this
>binary
>or one of the libraries it was linked against is corrupt, improperly
>built,
>or misconfigured. This error can also be caused by malfunctioning
>hardware.
>We will try our best to scrape up some info that will hopefully help
>diagnose
>the problem, but since we have already crashed, something is
>definitely wrong
>and this may fail.
>
>key_buffer_size=0
>read_buffer_size=33550336
>max_used_connections=0
>max_connections=512
>threads_connected=0
>It is possible that mysqld could use up to
>key_buffer_size + (read_buffer_size +
>sort_buffer_size)*max_connections = 4192252 K
>bytes of memory
>Hope that's ok; if not, decrease some variables in the equation.
>
>thd=(nil)
>Attempting backtrace. You can use the following information to find
>out
>where mysqld died. If you see no messages after this, something went
>terribly wrong...
>Cannot determine thread, fp=0x483466a8, backtrace may not be correct.
>Stack range sanity check OK, backtrace follows:
>0x81356bf
>0x47f420
>0x4f40fc59
>0x8348a4c
>0x834b141
>0x832f3a3
>0x836af8d
>0x8288364
>0xa4fbd4
>0x5684fe
>New value of fp=(nil) failed sanity check, terminating stack trace!
>Please read http://dev.mysql.com/doc/mysql/en/Us...ack_trace.html
>and follow instructions on how to resolve the stack trace. Resolved
>stack trace is much more helpful in diagnosing the problem, so please
>do
>resolve it
>The manual page at http://www.mysql.com/doc/en/Crashing.html contains
>information that should help you find out what is causing the crash.
>070309 10:10:33 mysqld ended
>
>************************************************* **********
>
>When resolving the stack dump I get this:
>
>resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack
>0x81356bf tab_gbk_uni0 + 33407
>0x47f420 (?)
>0x4f413c59 _end + 1193789273
>0x8348a4c _end + 1774924
>0x834b141 _end + 1784897
>0x832f3a3 _end + 1670819
>0x836af8d _end + 1915533
>0x8288364 _end + 986724
>0xa4fbd4 (?)
>0x5684fe (?)
>
>************************************************* **********
>
>Unfortunately this doesn't tell me much. It seems to be an InnoDB
>problem, since MySQL crashes when reaching 99% in the InooDB check...
>But what can i do to solve it?
>
>Any help is appreciated
>
>
>/GoL