Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The database is corrupt. #67

Open
mrexodia opened this issue Nov 20, 2020 · 20 comments
Open

The database is corrupt. #67

mrexodia opened this issue Nov 20, 2020 · 20 comments

Comments

@mrexodia
Copy link
Contributor

When restarting Windows 10 2004 I think something goes wrong with handling of the shutdown/restart event.

I get the following message on startup:

The database is corrupt. Please restore the last backup (ProcrastiTracker makes these frequently), and try again. See Procrastitracker start menu for database location. Exiting program..

When asked if I want to restore the database and click 'Yes' it fails and when I go to the %appdata% folder I see:

image
@aardappel
Copy link
Owner

Not sure why it corrupted, possibly it restarted while in the middle of writing.

Rename db.PT to db_corrupt.PT, and then db_TEMP.~PT to db.PT, then run again.

If it still doesn't work, try renaming db_BACKUP_2020_11_12.PT to db.PT. Sadly in that case you'll lose a week of stats.

@mrexodia
Copy link
Contributor Author

mrexodia commented Nov 20, 2020 via email

@aardappel
Copy link
Owner

Hmm.. I have no idea how that is possible. It saves the DB every 10 minutes. So no restart pattern should be able to interfere with that every time I'd think.

@mrexodia
Copy link
Contributor Author

I guess I just got unlucky multiple times in a row then 🤷‍♂️. I do not reboot often at all and every time I tested I reproduced the issue, but when I tested again today everything was working fine again.

There is a dialog that offers to recover the database and that reproducibly fails to restore the database though (likely because it's trying to restore that 0 byte .TMP file instead of the latest backup).

@mrexodia
Copy link
Contributor Author

The issue kept happening unfortunately and I discovered that the application silently dies in the background when leaving the PC on for a longer period of time. I couldn't find any crash dumps but I will write some application to monitor the state over time to try and narrow down the issue.

In every case the database ends up being corrupted unfortunately 🙁

@aardappel
Copy link
Owner

Hmm, that sounds like something is causing it to crash while its writing the DB specifically.

I could add some functionality to add a crash log, and/or send you a debug version of the program?

@mrexodia
Copy link
Contributor Author

mrexodia commented Nov 27, 2020 via email

@aardappel
Copy link
Owner

Oh if you can compile it yourself you can run it from VS (F5), so you could get the debugger to hit, that be even more valuable of course (see both a stack trace and variables).

When it "silently dies", the PC keeps running? no reboot?

I don't think programs do crash dumps unless they are programmed to do so, same with adding meaningful stuff to the event viewer.

Not sure what you mean by "just in time debugger". recommend running it under VS.

What I mean with "I could add some functionality to add a crash log" is code that catches exceptions and writes a stack traces to a log file as it crashes. That is mainly useful when NOT running under the debugger though.

@mrexodia
Copy link
Contributor Author

When it "silently dies", the PC keeps running? no reboot?

Yeah my PC keeps running, but it appears as though sometimes random applications are terminated (and apparently sometimes corrupting data).

You can configure Windows to automatically save crash dumps for specific applications: https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

The just-in-time debugger is an application that gets called by werfault.exe if an application crashes. Visual Studio itself can be configured as such (but I use JitMagic to allow me to choose a debugger). Here you can find more information: https://docs.microsoft.com/en-us/visualstudio/debugger/debug-using-the-just-in-time-debugger?view=vs-2019

@aardappel
Copy link
Owner

Ah yes, this "just in time debugging" I do have on, didn't know it was called that :)

Either way, let me know if you need any help from my side tracking this down.

@aburke20
Copy link

I'm actually having a very similar issue.

I'm away from home and using a laptop that every week or so crashes to BSOD.

The issue is that it somehow "corrupts" every single "backup" that Procrasti supposedly made. I get that it might corrupt the latest one, but how do none of them work? It's making me start from scratch every time the laptop crashes.

Wouldn't that be an issue with the way it is writing the backups, since they get corrupt where they are new or old, being written or written days ago ?

Any ideas as to a solution ?

@mrexodia
Copy link
Contributor Author

mrexodia commented Dec 12, 2020 via email

@aardappel
Copy link
Owner

The way it saves is that it first writes to db_TEMP.~PT. If it would crash during writing, then only this file should ever be corrupt, and all other files are not touched.

Only once this file has succesfully been written, does it:

  • if this is the first save this reboot, copy db.PT to a filename with a date in it.
  • delete db_BACKUP.PT
  • rename db.PT to db_BACKUP.PT
  • rename db_TEMP.~PT to db.PT

As you can see, that is a very "safe" sequence, exactly intended to avoid any file problems.

Also, the copy/delete/rename actions are the built-in Windows routines, not anything custom, i.e. it should have the same effect as when using these commands from the command-line for example.

So the fact that db.PT is 0 bytes in @mrexodia's screenshot is thus quite weird: what would have needed to happen for this is one of these two:

  • db_TEMP.~PT was "succesfully" written in the sense that PT didn't crash, but somehow the bytes didn't arrive on the disk anyway. OR:
  • The rename action (Windows function MoveFileA) screwed up.

I do see that the actual writing of the bytes does no error checking beyond seeing if the file can be opened. That should normally not matter, as it typically only fails if the disk is full or other rare circumstances. But I could add it just to help tracking these issues down.

procrastitracker/src/nodedb.h

Lines 2 to 37 in acc93cc

void save(bool filtered = false, char *givenfilename = NULL) {
static int firstsave = TRUE;
gzFile f = gzopen(givenfilename ? givenfilename : databasetemp, "wb1h");
if (!f) panic("PT: could not open database file for writing");
wint(f, FILE_FORMAT_VERSION);
wint(f, 'PTFF');
wint(f, MAXTAGS);
loop(i, MAXTAGS) {
gzwrite(f, tags[i].name, sizeof(tags[i].name));
wint(f, tags[i].color);
}
wint(f, minfilter.ival * 60);
wint(f, foldlevel);
loop(i, NUM_PREFS) wint(f, prefs[i].ival);
root->save(f, filtered);
gzclose(f);
if (!givenfilename) {
lastsavetime = GetTickCount();
// only delete previous backups if we have a database
if (GetFileAttributesA(databasemain) != INVALID_FILE_ATTRIBUTES) {
if (firstsave) {
SYSTEMTIME st;
GetLocalTime(&st);
char lrun[MAX_PATH];
sprintf_s(lrun, MAX_PATH, "%sdb_BACKUP_%d_%02d_%02d.PT", databaseroot, st.wYear,
st.wMonth, st.wDay);
DeleteFileA(lrun);
CopyFileA(databasemain, lrun, FALSE); // backup last saved of last run once per day
firstsave = FALSE;
}
DeleteFileA(databaseback);
MoveFileA(databasemain, databaseback); // backup last saved
}
MoveFileA(databasetemp, databasemain);
}
}

@aardappel
Copy link
Owner

Ok, I added stricter error checking on writing: 574bb8d
This should help with tracking these problems down, and should avoid the file being copied if there was any problem writing the file.

You'll now get one of these 3 errors:

  • could not open database file for writing - file wouln't even open.
  • write failed while writing database to disk - error while writing bytes.
  • could not finish writing database file - error while closing file. This is possible, since it's using zlib to write compressed files, so if there's a memory problem after PT is done writing all bytes but before zlib is done compressing all of them, this one may fail.

@aburke20
Copy link

aburke20 commented Dec 13, 2020 •
edited
Loading

Thanks so much for all this. It doesn't seem to still not be telling me the specific reason for the error (I'm seeing the same message as always).

Also, I don't know if it's "intentional" or not, but anytime I try to merge a backup, and it turns out that the backup is corrupt, the app crashes ... so basically I'm having to: try to merge a backup, have it tell me it's corrupt, then app crashes, then open the app again. Rinse and repeat as I try out all the backups.

Here are my last backups (I don't think there is any personal info in here, right ?), in case it can help you figure out what is breaking on my end.

https://pixeldrain.com/u/E3KB5wg9

Again, thanks so much for the effort.

edit: also it says "merge", but does that actually merge two sets on databases, or is it just a "replace current database with another one chosen from the list of backups" ?

@aardappel
Copy link
Owner

@aburke20 what is the same message as always, and when do you see it?

Rather than merge a backup, the intention is that you rename a backup to db.PT and it will just try to load that. Merge should in theory also work, but it is a more complex operation. Not sure why that would crash. You're basically in a running PT that has an empty database and you're trying to merge and it fails? I will try to see if I can repro that.

I will also try with your files, thanks.

Yes, merge actually merges, not replace.

@aburke20
Copy link

aburke20 commented Dec 13, 2020 •
edited
Loading

-Same:
"The database is corrupt. Last available backup: XX/XX/XX. Do you want to restore it ?"

Select NO (since I know what that last backup is, and it isn't the last one: which would have way more tracking data)

-Then: "The database is corrupt. Please restore the last backup. Procrasti makes these frequently ... yada yada yada"

App closes, and I see now it was saying it would close, so it's not a crash (sorry for confusion)

-So no special distinction as to why it was corrupt (which I think is what your latest update was trying to do (unless I'm mistaken).

-Basically right now I would love to be able to merge the data (from what I gave you) from the db.PT, and the one labeled from 11/25, since both of those has the largest data set of the last month or so. If not, then my 11/25 is just fine, and I have gotten that one to work. Other working ones are like 12/08, 12/06, 12/04 ... I think.

Again, no idea why those specific dates are able to be renamed and used as the core db.PT, but not the others ... all the "corrupt days" might be when Windows crashed ?

I'm so sorry for all this, I know I'm basically asking you to help me get your program to work "better", by somehow not failing despite the root initiating cause being my entire laptop crashing to BSOD.

@aardappel
Copy link
Owner

@aburke20
I so desperately want to play that game too ;) <- as you can tell, I can read your db.PT file fine.

I was also able to load your largest backup file by renaming it to db.PT.

I did discover there was an actual bug in the merge function which I just fixed: 738cf1d

So merge should now work, however, you should still be better off renaming the last working .PT file to db.PT instead, unless you have 2 files from separate date ranges that you want to merge.

@aburke20
Copy link

Woohoo, look at us testing and debugging ! I'll try the update.

OK, tested the merge, and got exactly what I wanted !

It also seems like you fixing the merge got rid of the "database is corrupt" issue, because I just tried to merge them all and they all merge just fine as if there was never a corruption issue !

As for the game, just promise to go into it with mild to moderate expectations.
I don't know if you've heard, but it turns out the game can be a little bit of a let down if you were too excited about what it has to offer. Main story is still very gripping, but overall the game world needs a little work.

Hope you can play it soon !

@aardappel
Copy link
Owner

That's good to hear!

Yeah, I am actually delaying playing it a bit because of all the bugs.. and see if I can get a 3080 beforehand ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mrexodia @aardappel @aburke20