FlexRAID on WHS

micksh wrote:

npht wrote:
Does this mean you recommend against using FlexRAID with WHS at this time? If it is an issue with the current release v1.2 I'd rather not risk hosing things up!

I've been using FlexRaid over WHS for about half a year. WHS never moved files across the drives except one case that happened about a month ago. I regularly run rsynch and if I don't change files it says that no data change is detected.
At that time when WHS moved files I think it moved MyMovies metadata to the same drive as the movie file it was related to. Most of the files were hidden so impact to FlexRaid was minimal, few kilobytes. Nothing like that happened since so I consider it an exception to the rule.
I think it is pretty safe even now (especially if you don't run MyMovies for WHS) but of course FlexRaid 1.3 will be very welcomed.

If I understand the implications correctly, the only risk comes if I actually need to restore data (from lost DRU) and the tombstones are at that point incorrect... does that simply mean the restored data won't be recoverable or does it mean WHS will be corrupted? I'm not sure I understand the implications of tombstone pointers being incorrect.

Thanks for your insight. It sounds as if it's a pretty small risk even in 1.2 - but does exist. The way I look at it is if I go ahead and set up 1.2 on my WHS (with new parts) the likelihood of me losing a disk before 1.3 comes out should be small. Even if i did lose a disk the chance of a restore being screwed up is also small... so together it seems like a small risk and even then I'm still only risking the one drive.

That or I just run backup in WHS on all media until v1.3 comes out. I can probably do that until I run out of disk space...

This message was edited 1 time. Last update was at 17/10/2009 20:50:20

No, tombstones don't matter in this case. The risk is that FlexRaid parity data will not be valid for some (small?) part of your data. If one of your drive dies you will be able to recover most (if not all in some cases) of your data anyway.
In example I provided parity would be incorrect for a few kilobytes. Since I mostly protect movie files chances are that the drive restore would lead to a few seconds of corrupted picture in one movie. If parity data refers to software archive on the drive that failed, the restored archive may be unreadable.

Yes, I think the risk is pretty small (I protect important documents and photos in WHS duplication outside FlexRaid anyway so for me the risk is negligible) but some risk does exist and version 1.3 is supposed to eliminate it.

So basically, WHS will not be corrupted. The only implication is that if your HDD fails and you restore it you may lose some part of your data. No problems in normal operation. The size of the data you will not be able to restore depends on size of data WHS moves since last rsynch before drive death.

The single case I experienced in 6 months amounts to few kilobytes of data I would have lost if one of my drives failed in short period between two rsynchs one month ago. Very small chance.

This message was edited 1 time. Last update was at 17/10/2009 21:53:00

micksh wrote:No, tombstones don't matter in this case. The risk is that FlexRaid parity data will not be valid for some (small?) part of your data. If one of your drive dies you will be able to recover most (if not all in some cases) of your data anyway.

Thanks for trying to explain this! This is where I get confused... I setup my parity disk (PPU) outside of the WHS pool so I don't see how WHS would mess with the parity at all. If I understand tombstones (lol, probably I don't) they are 4kb files that the OS creates to point at where files/data are. So lets say we synch with FlexRAID and create parity. After that point WHS decides to move stuff around... a disk (DRU) is lost... I go and restore that data with FlexRAID but the tombstones are now incorrect because some of the data I restore isn't where WHS "thinks" it is. I know something is wrong with my example because during restore FlexRAID restores to a share - so it would seem those tombstones would be created during the restore. Hmm, seems the only issue would be with orphaned tombstones that no longer point to correct files/data because of the lost disk.

I know I'm still missing something.

This message was edited 3 times. Last update was at 18/10/2009 01:06:04

You don't need to worry about tombstones if you only care about your data. Tombstones are pointers for WHS to know where you data is so WHS drive extender would know where to find it. You can read your data from any your particular drive without WHS. It's still regular NTFS file system.

The only reason we talk about tombstones is to consider a case when WHS system drive fails. Then currently (without feature of FlexRaid v. 1.3) you would have to reinstall WHS in order WHS to see your data. This is the way Microsoft recommends now if your WHS system drive fails. You will not lose your data, just have to go through long process of WHS reinstall to discover your data again. And then reinstall your programs/recreate users.

FlexRaid 1.2 takes care about your data, protects it from data drive failure. All tombstone integrity is taken care of in restoration process described here and FlexRaid 1.2 does it just fine.
FlexRaid 1.3 goes a step beyond, it will also protect tombstones. This is relevant only in case of WHS system drive failure. In this case you would restore WHS system from image (Acronis, Paragon or other HDD image backup program) and then restore tombstones with FlexRaid 1.3 so you don't have to reinstall WHS.

npht wrote:I setup my parity disk (PPU) outside of the WHS pool so I don't see how WHS would mess with the parity at all.

Version 1.2 of FlexRaid expects data to be on particular drives when it tries to restore the data after drive failure. In 99.9% of the cases the data will be there as expected. (I think my case is the only one reported case when WHS moved files between the drives. I will look more closely at this, I save all drive file listings and it never happened since. I don't know why it did this as my drives are not close to being full. You can expect WHS to move files when there is less than 20MB free on some drive)

If WHS moves the data from drive A to drive B because of its own sudden reasons Flexraid version 1.2 will not find it because it assumes data used to calculate parity to be on particular DRUs when it restores data. Version 1.3 will find it because it is more flexible.

npht wrote:I know I'm still missing something.

I think so, yes

Sorry, it is difficult for me to describe a whole picture in a few words. There is material on this site that tells everything.

micksh wrote:You don't need to worry about tombstones if you only care about your data. Tombstones are pointers for WHS to know where you data is so WHS drive extender would know where to find it. You can read your data from any your particular drive without WHS. It's still regular NTFS file system.

The only reason we talk about tombstones is to consider a case when WHS system drive fails. Then currently (without feature of FlexRaid v. 1.3) you would have to reinstall WHS in order WHS to see your data. This is the way Microsoft recommends now if your WHS system drive fails. You will not lose your data, just have to go through long process of WHS reinstall to discover your data again. And then reinstall your programs/recreate users.

I very much appreciate your comments and for taking the time to enlighten me. I have read quite a bit on these forums and on WHS forums but you filled in the key concept I was missing. I think it makes much more sense now!

I plan to install WHS on a (mobo based) RAID 1 set so the OS is on a mirror. That should pretty much eliminate (or nearly so) worries about having to reinstall WHS or related tombstone issues. I'll have the WHS pool (for media/data) on SATA controllers with a disk set aside from the pool for parity. I'll use FlexRAID for parity and have all the benefits of WHS. That should provide a very fault tolerant and robust NAS.

Thanks again for all your help!

Hope Brahim is less busy next week (and weekend?) and go a bit forward with this.

npht wrote:

npht wrote:

Brahim wrote:Not an issue in 1.3.
The risk of WHS moving data around at inadvertent time is handle correctly starting with 1.3.

Does this mean you recommend against using FlexRAID with WHS at this time? If it is an issue with the current release v1.2 I'd rather not risk hosing things up! Please advise, also if you know when you're thinking of releasing 1.3 I'm sure everyone would love to know!

Thanks, this is a very cool software!

Yes, wait until v1.3.

You could still use v1.2 for now and then use v1.3 if you ever have issue with restoring using v1.2.
Version 1.2 will tell you about any issue restoring.
The only thing that changes in v1.3 versus v1.2 in regard to WHS is how the restore process is handled.
Nothing about the RAID maintenance itself has changed.

Brahim wrote:

npht wrote:

npht wrote:

Brahim wrote:Not an issue in 1.3.
The risk of WHS moving data around at inadvertent time is handle correctly starting with 1.3.

Does this mean you recommend against using FlexRAID with WHS at this time? If it is an issue with the current release v1.2 I'd rather not risk hosing things up! Please advise, also if you know when you're thinking of releasing 1.3 I'm sure everyone would love to know!

Thanks, this is a very cool software!

Yes, wait until v1.3.

You could still use v1.2 for now and then use v1.3 if you ever have issue with restoring using v1.2.
Version 1.2 will tell you about any issue restoring.
The only thing that changes in v1.3 versus v1.2 in regard to WHS is how the restore process is handled.
Nothing about the RAID maintenance itself has changed.

I appreciate the feedback. Of course the last time I want issues is during restore! So, I guess the point is to minimize the risk of such problems. Based on the explanation of the risks by micksh it sounds as if the only real concern is losing the WHS OS - I'm using a RAID 1 mirror for the WHS OS so that shouldn't be a major concern.

I'm actually in the process of setting up the WHS/FlexRAID today using v1.2. I'll look forward to the release of v1.3 as well!

Thanks for your hard work

This message was edited 2 times. Last update was at 26/10/2009 12:24:57

The issue with WHS moving data around is not taken care of by mirroring the OS.

There were two issues being discussed.
So, let me clarify this.

Issue 1 - WHS moving data around after an rsynch:
Prior to v1.3, if WHS moved data around after an rsynch and you needed to recover some data, there was a change that the restore would fail since the original files would not be where FlexRAID expected them to be.
WHS tends to move the data across the drives from time to time as a way to balance it.
In version prior to 1.2, my recommendation was to disable that behavior in WHS (i.e., configure WHS to never move the data around).

Issue 2 - Tombstones:
WHS uses tombstones to create its consolidated view.
If a drive fails and you remove the drive from the pool, WHS will also remove all tombstone references pertaining to that drive.
When data is restored to the original drive directly (bypassing the WHS consolidated view), WHS does not restore the tombstones that it deleted.
WHS does not like to work with existing data and expects all write activities to happen through the view (even data restores).

The problem with restoring through the view is that WHS will not necessarily put them where they were originally.
It will treat the data being written through the view as new data and place it based on its data management algorithm.

Starting with v1.3, FlexRAID allows you to restore tombstones.
As such, users can restore the data to the original locations and then recreate the tombstones.

Restoring tombstones is an important feature beyond restoring a FlexRAID drive.
Even non-FlexRAID users will benefit from that feature.
The issue is that, if you lose your OS drive, WHS forces you to do a full install in order for it to recreate up the tombstones.
If you have a backup of your OS (using Norton Ghost, Drive Image, etc.), that backup won't work in restoring the tombstones and your tombstones will be out of sync.
With FlexRAID, you can now restore your OS from backups and restore/sync up the tombstones afterward.

So, restoring tombstones is not a FlexRAID issue but rather a WHS issue that FlexRAID has a solution for.

I don't want to distract you from your work so I'll make it real quick.

Brahim wrote:In version prior to 1.2, my recommendation was to disable that behavior in WHS (i.e., configure WHS to never move the data around).

I'm assuming you still recommend this disabling of moving data around in v1.2.

Brahim wrote:
Issue 2 - Tombstones:
WHS uses tombstones to create its consolidated view.
If a drive fails and you remove the drive from the pool, WHS will also remove all tombstone references pertaining to that drive.
When data is restored to the original drive directly (bypassing the WHS consolidated view), WHS does not restore the tombstones that it deleted.
WHS does not like to work with existing data and expects all write activities to happen through the view (even data restores).

I'm patient and willing to wait for v1.3 but I'm just trying to get a grip on v1.2/WHS and risk... since that's what using FlexRAID is all about. Do you have an idea about how likely restore issues are with v1.2/WHS (assuming the moving of data is turned off)? If the risk is high I just assume use folder duplication until v1.3 is out since the entire point of the NAS is to provide a fairly robust fault-tolerant configuration.

Thanks again

The risk is eliminated once you configure WHS to not move data around.
So yes, for v1.2, the recommendation is to configure WHS to not move the data around if possible.

Even if WHS is not configured to not move the data around, the risk is frankly very small since WHS does it infrequently.
Not only that, even without v1.3, there is a way (albeit a potentially tedious one) to get the moved files back where they were.
The restore process in v1.2 will log out all the moved files.
All it would be left to do is track each file, move them back, and rerun the restore process.

Nevertheless, the better solution would be to use v1.3 to do the recovery if you ever have an issue doing it with v1.2.

how you configure WHS to not move data around?

This message was edited 1 time. Last update was at 26/10/2009 17:32:19

Brahim wrote:You could still use v1.2 for now and then use v1.3 if you ever have issue with restoring using v1.2.

I've been holding off on installing FlexRAID until v1.3 was ready, but it sounds like there's no reason to wait (and I'd prefer to have some protection sooner rather than later). Am I correct in understanding that the "issues" with 1.3 that are currently keeping it in a beta stage are related to FlexRAID View, as opposed to the base FlexRAID parity function? If so, should someone using WHS (who therefore doesn't really have a need for FlexRAID view) install the 1.3 beta, or still stick with 1.2? If we should stick with v1.2 for now, will everything have to be set up again when 1.3 is final, or does all the previous setup get retained? Or are there differences to the setup due to the changes in v1.3?

I also second NLS's question about configuring WHS to leave the data alone. I didn't think there was really any way to do that.

Thanks!

dscline wrote:

Brahim wrote:You could still use v1.2 for now and then use v1.3 if you ever have issue with restoring using v1.2.

I've been holding off on installing FlexRAID until v1.3 was ready, but it sounds like there's no reason to wait (and I'd prefer to have some protection sooner rather than later). Am I correct in understanding that the "issues" with 1.3 that are currently keeping it in a beta stage are related to FlexRAID View, as opposed to the base FlexRAID parity function? If so, should someone using WHS (who therefore doesn't really have a need for FlexRAID view) install the 1.3 beta, or still stick with 1.2? If we should stick with v1.2 for now, will everything have to be set up again when 1.3 is final, or does all the previous setup get retained? Or are there differences to the setup due to the changes in v1.3?

I also second NLS's question about configuring WHS to leave the data alone. I didn't think there was really any way to do that.

Thanks!

I did some research and found some people have diabled the WHS storage manager service to stop balancing... but there's also addons that do the same thing... not sure what is the "best" way to do it for our purposes.

npht wrote:I did some research and found some people have diabled the WHS storage manager service to stop balancing... but there's also addons that do the same thing... not sure what is the "best" way to do it for our purposes.

Hmmm, are there any caveats to that though?

interesting - didn't know of such a plugin

also I am a bit worried about disabling a seemingly vital WHS service as we don't know what else it does

if you find the time please point us to the plugins you found

dscline wrote:

Brahim wrote:You could still use v1.2 for now and then use v1.3 if you ever have issue with restoring using v1.2.

I've been holding off on installing FlexRAID until v1.3 was ready, but it sounds like there's no reason to wait (and I'd prefer to have some protection sooner rather than later). Am I correct in understanding that the "issues" with 1.3 that are currently keeping it in a beta stage are related to FlexRAID View, as opposed to the base FlexRAID parity function? If so, should someone using WHS (who therefore doesn't really have a need for FlexRAID view) install the 1.3 beta, or still stick with 1.2? If we should stick with v1.2 for now, will everything have to be set up again when 1.3 is final, or does all the previous setup get retained? Or are there differences to the setup due to the changes in v1.3?

I also second NLS's question about configuring WHS to leave the data alone. I didn't think there was really any way to do that.

Thanks!

Yes, the only thing keeping v1.3 in RC stage is FlexRAID-View.

Also yes, uninstalling one version and installing another version is painless (at least, when not using FlexRAID-View).
At the very worst, you will need to redo your log settings if you had changed them from the default.
No reboot needed, and it should take less than 2 minutes to do the uninstall + install.

Also, RC releases are only good for 90 days.

That said, I do understand the position of someone wanting to only stick to final releases.
There is absolutely nothing wrong with that.

Ok, I decided to jump on 1.3

But now I seem to be having some learning curve issues:

I currently have only two drives in the pool, plus the "D:/" on the primary data partition. I have four shares I want to protect. Due to the way WHS has split the data up, not all three drives have all four of these folders, and I see no way in the WebGUI to manually create a path that doesn't exist. Do I understand the tutorial correctly in that I need to first manually create the folders in Windows file manager? Forgive me for such a dumb question, I just want to make sure before I start messing with the way WHS does things.

I also seem to be having issues creating subsequent DRUs. I can create the first one, and add multiple paths to it, but when I click the little green + icon to add a new DRU, all that happens is the paths in the first DRU disappear. If at that point I try to "fix" it by dragging a path back to the DRU, they all reappear (plus the duplicate I just dragged over). What am I missing here?

1. Yes, use Window explorer to create the missing folders

2. Make sure to use Firefox for the WebUI.
Any other browser will have issues.