How Does flexRAID Gaurd Against Gradual Drive Failure?

This message was edited 1 time. Last update was at 12/01/2010 10:51:03

The rollback feature will only capture the exact number of bytes that are to be changed.

So, if 1000 parity files are affected each with only one byte, the rollback data will only be 1000 bytes and not the sum size of the parity files.

[Edit] Also, even having the metadata stored with the rips should not amount to anything near 200GB.
My MyMovies metadata (albeit version 1) is only 200-300MB.
I also have it setup to store the metadata separately and not with the movies.
Not sure about the latest versions, but I am pretty sure you can change your configuration to store the metadata separately like in version 1.

This message was edited 1 time. Last update was at 12/01/2010 10:48:04

Hmmm, then there is something wrong, because that's not how it's working on mine. They are all exactly the same size, about 2GB, IIRC. I would say at least the first 15 minutes of an rsynch for me is just making the rollback files.

EDIT: but wouldn't you WANT to store the whole file anyway? I thought the whole point of the rollback was to have a back-up if an error occured re-writing the updated parity file. If you only backed up the changed part, how do you recreate the original if it got corrupted during the update?

dscline wrote:Hmmm, then there is something wrong, because that's not how it's working on mine. They are all exactly the same size, about 2GB, IIRC. I would say at least the first 15 minutes of an rsynch for me is just making the rollback files.

EDIT: but wouldn't you WANT to store the whole file anyway? I thought the whole point of the rollback was to have a back-up if an error occured re-writing the updated parity file. If you only backed up the changed part, how do you recreate the original if it got corrupted during the update?

The next time your run rsynch, take a clip of the process and sent it to me.

As stated, I capture what is to be changed and know how to restore it (think incremental backup instead of full backup).
It is simple in concept but complex in implementation, and it works.

Ok, by "clip", do mean something from the log file, or a screen cap of the rollback files?

dscline wrote:Ok, by "clip", do mean something from the log file, or a screen cap of the rollback files?

Screen cap & logs.

Is the rollback feature only available in the Command Prompt client? I don't see it in the WebGui.

nsainfreek wrote:Is the rollback feature only available in the Command Prompt client? I don't see it in the WebGui.

On the WebUI, you have to configure a URU (Undo Recovery Unit).
So, you have DRU, PPU, and URU.

And you also have to set "enableUndoOperation" in the runtime properties. IIRC, you have to expand that section towards the top of the screen where you would normally initiate an rsynch, validate, etc.

Thanks guys! I'll give that a shot when I get home today.

Brahim wrote:

dscline wrote:Ok, by "clip", do mean something from the log file, or a screen cap of the rollback files?

Screen cap & logs.

Sorry about being so late to reply back on this. I just haven't had the time to do a full test on it. But I do have a partial report, which gave me results I didn't expect, so perhaps I've misunderstood how it works.

As I mentioned previously, I typically leave the MyMovies service turned off to prevent these constant updates, and only turn it on when I rip new movies. But for the purpose of testing this, I just turned on the service for about 15 minutes to cause some updates, then ran an rsynch. It resulted in a single 1 GB parity file in the roll back folder until the rsynch completed.

First off, my memory was off, I guess the default parity size is 1GB, not 2. It did cause a "full size" undo file, but only one of them was created as opposed to the 50+ that I remember happening in the past. The primary difference, of course, is that there was no significant new data for this rsynch (no new rips), only some changed data.

So I guess my understanding of the rollback is off? I thought the rollback files were just backups of the old parity data, and therefore only CHANGED parity files would cause rollback files. But now my perception is that when undo is enabled, even new parity files are put in the rollback location and then moved to the normal parity location once they are all done. Is that correct? That would explain my perception of small changes causing a lot of rollback space... when I sit down and rip movies, I might do 30+ in one night. So I normally have a LOT of new data, plus whatever small changes there are from the metadata updates. Since I THOUGHT that the rollback only included changed parity files, I thought all those rollback files were only due to the changes, not the new files.

Is my (new) perception correct? Essentially, with undo turned on, any new (as well as changed) parity files first go to the undo location? I'll do the screen cap & logs the next time I actually rip, but I didn't bother this time since I didn't get the page full of rollback files like I had previously when doing new rips.

I guess I will need to see a screen caps or sceenshots of some sort showing what you are talking about.

The rollback data size will/should always match the process size, which is how much data has changed.
Anything else is a bug.
So, file a bug report with your evidence.

Brahim wrote:The rollback data size will/should always match the process size, which is how much data has changed.
Anything else is a bug.

I'm not sure there's even a problem, probably just a mis-perception on my part. I have multiple drives in my system, so it's entirely possible that what I'm seeing is simply due to drives being "out of balance". If I have one drive that has 750GB of data, and another that has 500GB of data, then perhaps what is happening is WHS is putting my new rips on the drive with only 500GB of data. In that case, the parity files that match up to that 250GB of space where the first drive maps up with nothing on the 2nd drive will be changed by the new data (the parity is now different).

So my initial caution of "changed data can cause more rollback space than you first expect" really needs to be "data that you consider "new" can really be a "change" from a parity perspective". I was seeing lots of parity files becoming "invalid" during validates due to small changes to metadata. I was also seeing large amounts of rollback files during a combination of these smal changes, and large amounts of new data. I was putting that together and assuming that small changes cause large amounts of rollback files, when in fact, it's probably the large of amount of new data that was causing that, due to the drives being "out of balance". If that makes sense.

Fair enough.

Yeah, it is always good to investigate when any thing appears even slightly off.
Keep monitoring your data and report back any anomaly.
It is far better to question and know with confidence what's going on with your data than be clueless and sorry later.

Brahim wrote:

gorman wrote:Could you clarify on the rollback feature? I have setup a rollback drive, just in case, but to be honest I haven't understood how and when to use it.

Thanks!

The rollback feature will undo all the changes made by an rsynch operation if necessary.
This is useful if a drive ever dies in the middle of an rsynch operation, and it is not able to complete.
In other words, it will bring your parity data back to a known state.

Is this rollback automated when flexraid determines a rsynch task failed or is their a command required to get your parity to rollback? What happens if you have power failure or a shutdown in the middle of a rsynch?

jrwalte wrote:

Is this rollback automated when flexraid determines a rsynch task failed or is their a command required to get your parity to rollback? What happens if you have power failure or a shutdown in the middle of a rsynch?

http://www.openegg.org/forums/posts/list/15/251.page#2605
http://www.openegg.org/forums/posts/list/15/251.page#2606

This message was edited 2 times. Last update was at 04/02/2010 15:29:23

Brahim wrote:

jrwalte wrote:

Is this rollback automated when flexraid determines a rsynch task failed or is their a command required to get your parity to rollback? What happens if you have power failure or a shutdown in the middle of a rsynch?

http://www.openegg.org/forums/posts/list/15/251.page#2605
http://www.openegg.org/forums/posts/list/15/251.page#2606

Thanks for the reply. I didn't word my post clearly enough. I understand how to enable the rollback, but how does the rollback function?

1) How does flexraid know when to rollback?
2) Do I need to run a command to tell it to?
3) What happens if there is a power failure during rsynch?

Yeah, I have the same question as jrwalte. I enabled the undo function, but should the rsynch fail I'm not sure how to use the undo data.

This message was edited 1 time. Last update was at 25/02/2010 16:37:33

nsainfreek wrote:Yeah, I have the same question as jrwalte. I enabled the undo function, but should the rsynch fail I'm not sure how to use the undo data.

The rollback is automatic.

There is no command to run.

It works this way:
1. FlexRAID checks whether the user has configured a URU
2. If so, it backups the data it is about the change (bit for bit). If it is changing one byte, one byte and one byte only will be backed up (not a whole parity file).
3. If the backup was successful, it goes along and commit the rsynch changes, else if fails the task.
4. If the backup was successful but the rsynch fails, it rollback the changes using the backup.
5. If the backup was successful and rsynch succeeds, it cleans up the backup data.

There is one more enhancement I will make, but it has a lesser priority.

Thanks for the explanation

If I was running a backup and halfway through I lost power. How would I get FlexRaid to recover from that? Would rsynching cause it to recognize that there is data in the undo folder and to restore that first?