corrupt datafile 2004-06-12 - By Terry Sutton
Re: corrupt datafileTim,
Well said on all points.
1. They (or at least we) would have logged a TAR if they had renewed it. :-(
2. The whole hot backup from Wed isn 't available, just the copy of the now-corrupted file. It seems that a current backup overwrites the previous backup, and the only reason the Wed copy of the corrupted file exists is that it couldn 't be copied (due to the corruption) during the backup. :-(
3. The reason that there aren 't copies going back is that they couldn 't spend money on disk space. As they tell me this I glance at the 120GB Iomega drive I use to backup my notebook. People gotta realize that they don 't need Sun arrays to keep some backup files around.
4. You 're also probably right about the export. I haven 't played with that yet, though I have told them it 's likely to be the only hope.
5. You 're right about them blaming themselves. We got called in to see if we could salvage anything. It 's great to pick up new clients, and it 's really great if we can be heroes, but then it 's depressing if you can 't bail them out. Even if it 's their fault, you feel bad for them.
Hopefully this will help someone else convince their management/clients of the importance of thorough backup strategies. They not necessarily expensive, and when they 're not used the expense can be astronomical.
Cheers,
Terry
Terry,
Hopefully they have logged a TAR with Oracle Support and are working it that way. It is possible that "dbverify " is returning the response that every block is corrupt simply because it cannot read the file-header information, not because the entire file is shredded. This file might have a few bytes out of place in the header, in which case it is possible that Oracle Support 's Data UnLoader (DUL) utility might be able to dump the data from the file. Otherwise, if every single block is corrupt, then even DUL probably would not be able to retrieve the data.
A "hot backup " from Wednesday is available but the archivelogs going back to that backup aren 't? Other datafile backups from the "hot backup " on Wednesday are not available? It seems that many, many things have gone wrong here. It would be worthwhile to dwell on why the other datafiles and the archivelogs are not available, so that this never happens again. This is a time for brutal honesty.
From what you describe, there are no options. This database is kaput and complete recovery is impossible. Only two "incomplete recovery " options remain: use DUL to pull data from the dead datafiles or use the 6-month-old export. And given the trail of mistakes, that export was probably not taken with CONSISTENT=Y or the database was not fully quiesced, meaning that it is probably almost useless too. So, DUL is probably the only possibility, and Oracle Support charges for that.
These folks should not blame anyone (or anything) but themselves. They blew straight through the many fail-safes that Oracle provides, so that any one of a thousand minor mishaps would have landed them in this exact same situation. It was just waiting to happen.
My sincere condolences...
-Tim
on 6/12/04 2:21 PM, Terry Sutton at terrysutton@(protected) wrote:
A new client has a serious corruption error. During a hot backup (not RMAN) Thursday morning it was discovered that a datafile was corrupted on the disk. When the database is started up, the following error occurs:
"ORA-01122 (See ORA-01122.ora-code.com): database file 5 failed verification check
ORA-01110 (See ORA-01110.ora-code.com): data file 5:
'/ora2/app/oracle/admin/dbn/data/dbn_data_01.dbf '
ORA-01251 (See ORA-01251.ora-code.com): Unknown File Header Version read for file number 5 "
When dbverify is run against this file, every block is indicated as corrupt, which isn 't surprising.
We have a copy of the datafile from Wednesday 's hot backup, but they don 't have the archivelogs necessary to sync this datafile with the rest of the database. And we don 't have the other datafiles from Wednesday 's backup, so we can 't just go back to the state the database was in then.
Any ideas on what they can do? I can 't think of anything (other than going back to an export they have from 6 months ago, which seems to be the only historical backup they have).
--Terry
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN " >
<HTML > <HEAD > <TITLE >Re: corrupt datafile </TITLE >
<META http-equiv=Content-Type content= "text/html; charset=iso-8859-1 " >
<META content= "MSHTML 6.00.2800.1400 " name=GENERATOR >
<STYLE > </STYLE >
</HEAD >
<BODY bgColor=#ffffff >
<DIV > <FONT face=Arial size=2 >Tim, </FONT > </DIV >
<DIV > <FONT face=Arial size=2 > </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >Well said on all points. </FONT > </DIV >
<DIV > <FONT face=Arial size=2 > </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >1. They (or at least we) would have logged a
TAR if they had renewed it. :-( </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >2. The whole hot backup from Wed isn 't available,
just the copy of the now-corrupted file. It seems that a current backup
overwrites the previous backup, and the only reason the Wed copy of the
corrupted file exists is that it couldn 't be copied (due to the corruption)
during the backup. :-( </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >3. The reason that there aren 't copies going back
is that they couldn 't spend money on disk space. As they tell me this I
glance at the 120GB Iomega drive I use to backup my notebook. People
gotta realize that they don 't need Sun arrays to keep some backup files
around. </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >4. You 're also probably right about the
export. I haven 't played with that yet, though I have told them it 's
likely to be the only hope. </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >5. You 're right about them blaming
themselves. We got called in to see if we could salvage anything.
It 's great to pick up new clients, and it 's really great if we can be heroes,
but then it 's depressing if you can 't bail them out. Even if it 's their
fault, you feel bad for them. </FONT > </DIV >
<DIV > <FONT face=Arial size=2 > </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >Hopefully this will help someone else convince
their management/clients of the importance of thorough backup strategies.
They not necessarily expensive, and when they 're not used the expense can be
astronomical. </FONT > </DIV >
<DIV > <FONT face=Arial size=2 > </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >Cheers, </FONT > </DIV >
<DIV > <FONT face=Arial size=2 > </FONT > </DIV >
<DIV > <FONT face=Arial size=2 >Terry </FONT > </DIV >
<BLOCKQUOTE dir=ltr
style= "PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px " > <FONT
face=Arial > <SPAN style= "FONT-SIZE: 12px " >Terry, <BR > <BR >Hopefully they have
logged a TAR with Oracle Support and are working it that way. It is
possible that ?dbverify? is returning the response that every block is corrupt
simply because it cannot read the file-header information, not because the
entire file is shredded. This file might have a few bytes out of place
in the header, in which case it is possible that Oracle Support?s Data
UnLoader (DUL) utility might be able to dump the data from the file.
Otherwise, if every single block is corrupt, then even DUL probably
would not be able to retrieve the data. <BR > <BR >A ?hot backup? from Wednesday
is available but the archivelogs going back to that backup aren?t? Other
datafile backups from the ?hot backup? on Wednesday are not available?
It seems that many, many things have gone wrong here. It would be
worthwhile to dwell on why the other datafiles and the archivelogs are not
available, so that this never happens again. This is a time for brutal
honesty. <BR > <BR >From what you describe, there are no options. This
database is kaput and complete recovery is impossible. Only two
?incomplete recovery? options remain: use DUL to pull data from the dead
datafiles or use the 6-month-old export. And given the trail of
mistakes, that export was probably not taken with CONSISTENT=Y or the database
was not fully quiesced, meaning that it is probably almost useless too.
So, DUL is probably the only possibility, and Oracle Support charges for
that. <BR > <BR >These folks should not blame anyone (or anything) but themselves.
They blew straight through the many fail-safes that Oracle provides, so
that any one of a thousand minor mishaps would have landed them in this exact
same situation. It was just waiting to happen. <BR > <BR >My sincere
condolences... <BR > <BR >-Tim <BR > <BR > <BR >on 6/12/04 2:21 PM, Terry Sutton at
terrysutton@(protected) wrote: <BR > <BR > </SPAN > </FONT >
<BLOCKQUOTE > <FONT face=Arial > <SPAN style= "FONT-SIZE: 12px " >A new client has
a serious corruption error. During a hot backup (not RMAN) Thursday
morning it was discovered that a datafile was corrupted on the disk.
When the database is started up, the following error
occurs: <BR > <BR > "ORA-01122 (See ORA-01122.ora-code.com): database file 5 failed verification
check <BR >ORA-01110 (See ORA-01110.ora-code.com): data file
5: <BR > '/ora2/app/oracle/admin/dbn/data/dbn_data_01.dbf ' <BR >ORA-01251 (See ORA-01251.ora-code.com):
Unknown File Header Version read for file number 5 " <BR >When dbverify is run
against this file, every block is indicated as corrupt, which isn 't
surprising. <BR > <BR >We have a copy of the datafile from Wednesday 's hot
backup, but they don 't have the archivelogs necessary to sync this datafile
with the rest of the database. And we don 't have the other datafiles
from Wednesday 's backup, so we can 't just go back to the state the database
was in then. <BR > <BR >Any ideas on what they can do? I can 't think
of anything (other than going back to an export they have from 6 months ago,
which seems to be the only historical backup they
have). <BR > <BR >--Terry </SPAN > </FONT > </BLOCKQUOTE > </BLOCKQUOTE > </BODY > </HTML >
|
|
