Questions on FLexraid view

Hi all- new here - first i should mention this is some great software!

I'm trying to mimic what Windows Home Server does by creating a "JBOD storage pool" using Flexraid view, but I'm having some issues and was hoping for some help.

If I have this viewconfig.txt:

DRIVE=E
UNIQUE=C:\Mount\Disk01;C:\Mount\Disk02;C:\Mount\Disk03
RESTRICT=
RESERVE=2GB
THREAD=5
REMOVABLE=FALSE

-E:\
|-*C:\Mount\Disk01
|-*C:\Mount\Disk02
|-*C:\Mount\Disk03

And I have a folder structure on the drives like:
C:\Mount\Disk01\Movies
C:\Mount\Disk02\Movies
C:\Mount\Disk03\Movies

In the view it creates 3 separate movie folders rather than merging their contents, e.g.:
E:\Movies
E:\Movies
E:\Movies

Is there a way to have flexraid view merge all the folder contents together and present me with 1 view?

Thanks in advance!

This message was edited 1 time. Last update was at 20/11/2010 00:56:42

Also I have a problem while testing viewconfigs when I use the cmd client to:
view stop
then
view start
it won't restart and instead I have to restart the computer.
(I'm using the latest flexraid version) - is anyone else having this issue?

[2010-11-20 17:54:03,596] ERROR: View handler initialization failure!
java.io.IOException: View handler initialization failure!
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)

webs0r wrote:Hi all- new here - first i should mention this is some great software!

I'm trying to mimic what Windows Home Server does by creating a "JBOD storage pool" using Flexraid view, but I'm having some issues and was hoping for some help.

If I have this viewconfig.txt:

DRIVE=E
UNIQUE=C:\Mount\Disk01;C:\Mount\Disk02;C:\Mount\Disk03
RESTRICT=
RESERVE=2GB
THREAD=5
REMOVABLE=FALSE

-E:\
|-*C:\Mount\Disk01
|-*C:\Mount\Disk02
|-*C:\Mount\Disk03

And I have a folder structure on the drives like:
C:\Mount\Disk01\Movies
C:\Mount\Disk02\Movies
C:\Mount\Disk03\Movies

In the view it creates 3 separate movie folders rather than merging their contents, e.g.:
E:\Movies
E:\Movies
E:\Movies

Is there a way to have flexraid view merge all the folder contents together and present me with 1 view?

Thanks in advance!

Maybe try something like this:

-X:\
|-<Movies>

-X:\Movies
|-*C:\Mount\Disk01\Movies
|-*C:\Mount\Disk02\Movies
|-*C:\Mount\Disk03\Movies

In this config "|-<Movies>" creates a virtual folder 'Movies', and then you define it's content.

Hope this works for you.

This message was edited 1 time. Last update was at 20/11/2010 13:25:53

Do as aay (!?) suggests.

You see FlexRAID-View doesn't do merges "implicitly" and needs an explicit definition of ALL POSSIBLE MERGES (including the ones in the SUBDIRS that are possibly needed). So even with the above, if you also have

"C:\Mount\Disk01\Movies\My Favorites"
and
"C:\Mount\Disk02\Movies\My Favorites"

...then you need to make an explicit definition to merge "My Favorites" too (and same for any possible "merge" that goes deep).

Hope you like it.

Don't ask why it has to be done like that, because our friend the author of FlexRAID believes that this is the right way (not considering the possibility to even have this as an option) as it would "create more problems than it solves" (pending a good example on this and in contrast to other "merge" solutions on various OSes). On the other hand if you still feel the need to ask why it is done like that, please do.

(sorry Brahim, had to brief him, didn't I?

)

Thanks aay and NLS.

I will do as you suggest. I'll probably end up simplifying my folder structure (make it flat) so that it is easier to manage rather than having my nice groupings.

Unfortunately I have to go to lunch so I can't play with my new server until a bit later

Maybe this will fix the blue screen problem as well - after I try the new config I will try to share again (I was getting a blue screen on the server whenever I tried to access the view share from a client) - but I didn't see anyone in the forum post about this before.

Brahim, I hope you would reconsider the merge functionality - this way it requires much more care from me - I would have to many more edits to the ViewConfig.txt whenever I add a drive and be much more conscious of where I put new files. In the other case I could just set it and forget it, add a line when I put a new drive in and let Flexraidview take care of the folder structure and where the files go!

Ahhh I get it now.

So if I want to mirror my WHS config I would have to make a really really big config file that would require a lot of maintenance if I want all subfolders "virtualised" across all drives. (if it would accept it)
I'd have to somehow set up excel to produce the ViewConfig and also need to script some way to create a whole bunch of folder creates whenever I add a new drive ... hmm

Or to simplify it you would just target the folders that are getting new data added to them, but you would have to be careful if you added something to another older folder that hasn't been properly setup. (Works better with static data)

Ok so question:

What happens if you have "spanned"
x:\videos\series across multiple drives, and one drive is getting really really full
then you are putting files into
x:\videos\series\someseries
which is on that drive that is getting really full
would it run out of room, or would it put it onto another drive that the parent folder is spanned to? Would it create the subfolder \someseries?

FYI

http://www.openegg.org/forums/posts/list/474.page

Hey guys so I'm kinda not getting anywhere now.

Apologies for the long post in advance!

I thought the View would be perfect but now I've got issues.

I've looked at Liquesce and that looks just about right but sharing doesn't work to 64 bit Windows clients.

I want to avoid WHS (the current fileserver) due to chkdsk running every 12 hours, outdated code base (I want 3Tb drive support) and general messiness if I have to restore files (and Server 2008 just seems cool!).

That Linux mhddfs looks great but I don't want to run Linux, it will take me too long to figure it out, plus I am likely to have have driver issues due to unsupported hardware.

I've considered running a script to generate a ViewConfig, but I have 1790 subdirectories that would need to be spanned across potentially 23 drives and I'm not confident Flexraid View would be able to gracefully handle this. Just the listing is 88 Kb. (Has anyone done this successfully? - I think I will try anyway - will let u know what happens once I figure out how to automate it)

I've considered identifying the subdirectories that get more frequently "added to". So I add maybe 200-400 gb of data per month to my "pool". Most of the quantity of subfolders is due to TV Series - I have \Videos\Series\Series Name\Series Season. So if I *just* target the ones that get more action I would span maybe 30 subfolders. But then I'm concerned when a drive fills up, anything could spill out - for example, adding a new Season to a Series that I haven't spanned. So I have to constantly be very conscious of any new folder action that might require me to change the Viewconfig.

I guess there are 2 issues:
1. I just don't know which future file is going to be added that will cause it to spill to the next drive, so I don't know which folder to span
2. I would have to span all the subfolders that I think I'm going to need to continue to add to in the future, but I don't know which they will be!
Actually, it's really the same issue... hehe

Seeking advice from you guys.... what do you think I should do??

I'm seriously leaning towards continuing WHSv1, but with Flexraid added for parity only. Just for the drive extender. And having to deal with chkdsk spinning up a zillion drives every 12 am and pm which would otherwise just keep sleeping

And potentially causing hiccups to file streaming (it certainly does hiccup while watching content via the box itself).

I would run far from WHSv1 (and to be honest even v2). I never looked back.

As for the rest, you just one of the users I described to Brahim when talking about implicit merges (or better: failed to describe).

...anyway...

I'm strongly considering moving from WHS to FRV for my drive pooling. But if I understand this conversation correctly, I will have to recreate (and maintain?) my entire directory structure in a txt file. This doesn't seem like a workable option for a large media collection. I have hundreds of folders, and stuff gets moved or added all the time. Is this really necessary, or am I not understanding the explanation?

This message was edited 1 time. Last update was at 24/11/2010 12:19:37

NLS wrote:I would run far from WHSv1 (and to be honest even v2). I never looked back.

As for the rest, you just one of the users I described to Brahim when talking about implicit merges (or better: failed to describe).

...anyway...

candre23 wrote:I'm strongly considering moving from WHS to FRV for my drive pooling. But if I understand this conversation correctly, I will have to recreate (and maintain?) my entire directory structure in a txt file. This doesn't seem like a workable option for a large media collection. I have hundreds of folders, and stuff gets moved or added all the time. Is this really necessary, or am I not understanding the explanation?

An auto merge feature will be mostly to support users migrating from WHS who do not want to merge back their data prior to using FlexRAID-View.
That feature (auto merge) is in the extreme back burner (if ever).
Essentially, I don't want to restrict my initial vision for the View with having to support legacy modes, and I certainly make no apologies for that.
If you want to use FlexRAID-View, you will have to adapt to its requirements or just use something else.
You have to be glad that it does not require you to format your drives (a la WHS and other solutions).

So, rather than wasting time discussing what will likely never happen (auto merge), we should probably start a thread discussing migration techniques from WHS to FlexRAID-View.
Anybody who tells you about having to constantly edit the View config file simply does not understand the product and need to spend more time doing so rather than spewing utter non-sense.
If it is too much trouble to merge back your data prior to using FlexRAID-View, then you should probably stick to WHS DE or look elsewhere as FlexRAID-View is not trying to address your particular problem.

I could easily create a script that will create a View config from drives that require auto-merging, but I really don't want to as I don't want users to think of FlexRAID-View in that concept.
Split folders should be kept at a minimum and managed exclusively by the View.

Again, I'm not 100% clear on what you're saying. What I want is essentially what DE does now - take a bunch of drives and pool them into one big virtual drive. Then dump all my data and organize it on that one large disk. What you appear to be telling me is that FRV doesn't work like that. I would have to create the directory structure on all the individual drives, then use FRV to make it "look like" one big drive over the network. If I wanted to create a new directory, and I wanted it to be able to span multiple drives, I would have to create that directory on each drive and add it to the config file.

If this is accurate, then FRV is far less useful than DE. It's really no more useful (and much more configuration-intensive) then libraries in Win7.

This message was edited 4 times. Last update was at 24/11/2010 15:44:25

No cadre you have confused things a bit, but I am sure Brahim will explain better (plus it is a discussion I don't like as I don't agree with his point of view on this, so I won't participate). FlexRAID-View will spill data under specific circumstances to other disks but unless you clearly define how, it will just use it's own hidden structure. Maybe good when you start from scratch (just maybe), but a nightmare when you have disks full of data. Anyway, Brahim can explain better and less biased (or at least biased from his side

).

The only thing I want to say is this (and what made me write this post):

An auto merge feature will be mostly to support users migrating from WHS who do not want to merge back their data prior to using FlexRAID-View.

...erm... in fact no. You just describe one case it would be like that (remember I for one DID take the time to move all my data around so that I would make a smaller viewconfig.txt). I wonder why can't you realize that sometimes (sizes involved) things HAVE to be like that (i.e. spread among disks) and I mean in considerable folder depth. And yes AFTER trying to "group" things as much as possible. Even if we talk about A SINGLE FOLDER that has to span, but this folder is... unlucky enough to be IN LEVEL FOUR depth, then this automatically means your system needs FOUR explicit merge definitions. As I said before, with GROUPED data, I can easily see scenarios with 100 config lines. Problem not just being those 100 lines, but how to manage this whole thing in case a small change needs 10 more explicit merges etc. I need a computer to do the dirty work for me, not me do the dirty work for my computer. I do not plan to administer my home server more than back when I was administering a Bank's network with numerous clusters, SANs and blades (and I doubt anybody disagrees).

...but since your mind is set on this, I don't have anything else to say on this. You possibly disagree with what the whole world thinks a "merge view" is (or should be), but since I don't have actual statistics to back me up (plus I don't develop any more - which is an art I hate to have lost and no time to recover), forget that I even said that.

I must consider it a small win that at least we got a "That feature (auto merge) is in the extreme back burner (if ever)."

(which is a shade of gray above total black)

EDIT: I feel the need to clear out that I do like FlexRAID (and waiting for -Live part), just not the -View and hope all these are understood for what they are: Constructive discussion between web friends.

Hey did u guys see this:

http://windowsteamblog.com/windows/b/windowshomeserver/archive/2010/11/23/windows-home-server-code-name-vail-update.aspx
MS is removing Drive Extender from WHSv2 "Vail"

Key feature - gone. I guess they realised they got the design totally wrong based on the MS Connect site (Vail feedback) and thought it was just better to release the product "on schedule" without it hehehe.

Brahim wrote:
Split folders should be kept at a minimum and managed exclusively by the View.

Hi Brahim,
So do you mean that I really should try this:
Merge the drive roots only, maybe copy recursive with subdir create on from my WHS in the view itself - and just let FRView software manage it?

I might give it a go... Actually I'd just teracopy each share root one by one.

Where does the metadata get stored in case of a drive failure - will FR know where files should belong in terms of folders, if a drive is destroyed?

I was looking into gentoo linux yesterday

candre23 wrote:Again, I'm not 100% clear on what you're saying. What I want is essentially what DE does now - take a bunch of drives and pool them into one big virtual drive. Then dump all my data and organize it on that one large disk. What you appear to be telling me is that FRV doesn't work like that. I would have to create the directory structure on all the individual drives, then use FRV to make it "look like" one big drive over the network. If I wanted to create a new directory, and I wanted it to be able to span multiple drives, I would have to create that directory on each drive and add it to the config file.

If this is accurate, then FRV is far less useful than DE. It's really no more useful (and much more configuration-intensive) then libraries in Win7.

No, you are very confused... and I blame NLS for this...

NLS wrote:
...

...erm... in fact no. You just describe one case it would be like that (remember I for one DID take the time to move all my data around so that I would make a smaller viewconfig.txt). I wonder why can't you realize that sometimes (sizes involved) things HAVE to be like that (i.e. spread among disks) and I mean in considerable folder depth. And yes AFTER trying to "group" things as much as possible. Even if we talk about A SINGLE FOLDER that has to span, but this folder is... unlucky enough to be IN LEVEL FOUR depth, then this automatically means your system needs FOUR explicit merge definitions. As I said before, with GROUPED data, I can easily see scenarios with 100 config lines. Problem not just being those 100 lines, but how to manage this whole thing in case a small change needs 10 more explicit merges etc. I need a computer to do the dirty work for me, not me do the dirty work for my computer. I do not plan to administer my home server more than back when I was administering a Bank's network with numerous clusters, SANs and blades (and I doubt anybody disagrees).

...but since your mind is set on this, I don't have anything else to say on this. You possibly disagree with what the whole world thinks a "merge view" is (or should be), but since I don't have actual statistics to back me up (plus I don't develop any more - which is an art I hate to have lost and no time to recover), forget that I even said that.

I must consider it a small win that at least we got a "That feature (auto merge) is in the extreme back burner (if ever)." (which is a shade of gray above total black)

Yep, you are right... I can't see it.
What I can see is a user having content in various places that can be aggregated in a single view... hence, why we have the merged view feature (logical organization).

Having more split folders than you have drives and not being able to merge those back is pure rubbish... just spend 5 seconds to think about it.
So please, let not confuse what you can't do and what you don't want to do.

Brahim wrote:
No, you are very confused... and I blame NLS for this...

Whoever is at fault (most likely myself), I'd still like some clarification. Starting from scratch, is it possible to use FRV to create a simple virtual JBOD from a collection of disks? Can I make one big virtual drive and add all my files and folders like with WHS, or do I have to create the directory structure in a config file?

candre23 wrote:
Whoever is at fault (most likely myself), I'd still like some clarification. Starting from scratch, is it possible to use FRV to create a simple virtual JBOD from a collection of disks? Can I make one big virtual drive and add all my files and folders like with WHS, or do I have to create the directory structure in a config file?

Yes.
http://www.openegg.org/forums/posts/list/290.page

This message was edited 1 time. Last update was at 25/11/2010 13:37:24

Brahim wrote:No, you are very confused... and I blame NLS for this...

Not a very cool comment.

I re-read the thread and I fail to see if I said anything not valid or how I might have confused our friend.

Anyway, I am sure he will sooner or later have a clear... view (no pun intended) on how the system works and I'd really like him to come back to this thread and point me wrong in whatever I wrote.

You just keep up with the good progress you seem to be making lately.

This message was edited 1 time. Last update was at 25/11/2010 13:59:12

(don't forget my post just above - since forum usually jumps way too forward when clicking for new posts)

BTW a quick example of my "rubbish" logic. I must probably be the only one on Earth with such a structure.

- D:\ 500GB
- E:\ 500GB
- F:\ 320GB
- G:\ 320GB
- H:\ 250GB
(let's assume we have the full of their claimed size - a total of 1890GB)

- Videos 720GB
- Audio 510GB
- Pictures is 260GB
- Misc 300GB
(a total of 1790GB)

...but...

- In "Videos" 650GB of the 720GB belong to "Movies" subfolder. The other 70GB are in another subfolder of "Videos", called "MiscVideos" with several sub-subfolders (like "Animation", "Adevertisements" etc.).

- In fact 580GB of the 650GB of "Movies" subfolder, is another subfolder is "HD" sub-subfolder. (I am not going deeper, but trust me, I could easily go and still be a VERY realistic example).

- In "Audio", 480GB are "MyWorks" and the rest 30GB is in another "Audio" subfolder, called "MP3".

- In "Pictures" there is a 200GB "RAWLibrary" subfolder and the other 60GB spread on several subfolders directly under "Pictures".

- In "Misc" there are several folders, but unfortunately again there is one big folder. It is 260GB of the 300GB. The path is "\Misc\Private\Backups\SecondaryServer\DataDrive". Why not directly under "Misc" or maybe separate share? But why DICTATE the user to organize his data as they want? So please, that is the path.

...Please Brahim (or someone else), let's do the exercise.
Let's assume a "virgin" setup (we can put anything wherever we want -let say because we restore all these from tape- without having to shift any pre-existing data).
Please... I am all... erm... eyes to see the proper viewconfig.txt. When someone provides me with this, I will post the next step (oups... a couple of specific folders just got larger and HAVE to "spill")... just to make it more interesting.

Thanks.

(in fact it would be a nice demo for people wanting to understand the power of view configuration file)