Understanding FlexRAID's Storage Pool Merge Modes

This message was edited 1 time. Last update was at 28/02/2011 21:56:19

Understanding FlexRAID's Storage Pool Merge Modes

FlexRAID (as of version 2.0 preview 6) supports several merge modes:

Explicit Merge:
This is the mode you are all familiar with.
This mode allows for a lot of flexibilities and really intricate configurations.
It allows you to treat your source data as a database and allows you to create Views into that data (no need to mess with symbolic/hard links).
Additionally, it provides the greatest energy saving feature.

Here is an example of an explicit merge configuration:

The first view you define has to be the root view (-driveLetter:\).
After you've defined the root view, you can then define its sub views.

A "-" specifies the containing folder (Ex: -K:\)
A "|-*" means, merge this path into the containing folder
A "|-+" means add this path as an element of the containing folder
A "|-<...>" defines a virtual folder (which is always a child of the containing folder)

A virtual folder only exists through FlexRAID-View and is used to virtually organize your data without cluttering your filesystem.

Auto Merge with Balanced Space Priority:
With this mode the configuration is automatic and there is no FlexRAID management folder to deal with.
This mode is very similar to what you are used to with WHS with the difference being in that it balances the disk space so that the drives are used evenly.

Auto Merge with Minimized Folder Split Priority:
As above, the configuration is automatic in this mode and there is no FlexRAID management folder to deal with.
Rather than balancing the data based on free space, however, this mode tries to keep folder splits to a minimum, which can cause certain drives to be used more than others.

Note:
The auto merge modes are not energy efficient as they frequently poll on the drives in the array (specially the Auto Merge with Balanced Space Priority).
When I get more time, I will see if can come up with something that changes that aspect.

Thanks.

Under explicit mode I know that virtual folders are placed in the hidden folder \_flxr_\r & one of the disks will have the folder structure replicated (not hidden).

With the automatic modes, does that mean I would have to migrate the tree structure back up to the root of the drive and un-hide the folders for the merge to work as expected?

Correct.
You might also need to change the folder attributes (back to normal) on the migrated folders (and sub-folders).
This is one of those things that could use a good automation script.
Any taker?

Note, such migration requires the parity to be totally recreated for a real-time RAID setup whereas it is just a quick update (metadata update only) for a snapshot RAID setup.
I entered a feature request to enable metadata update only for real-time RAID setups: http://bug.flexraid.com/view.php?id=24

Last, the auto modes also create _flxr_ folders, but won't use them except in a few transient situations. So, don't delete those if they are re-created when in auto mode.

This message was edited 1 time. Last update was at 02/03/2011 20:30:14

I installed the new release and chose Auto Merge with Minimized Folder Split Priority.

I manually moved my folders for 20 drives up to the root as suggested. The preview showed perfectly the setup I wanted to achieve, however when I actually published the configuration I have up to 20 copies of each folder at the root level as well as multiple folder duplicates inside. I'll post this on the bug tracker. Anyone else tried this yet?

nisemono wrote:I installed the new release and chose Auto Merge with Minimized Folder Split Priority.

I manually moved my folders for 20 drives up to the root as suggested. The preview showed perfectly the setup I wanted to achieve, however when I actually published the configuration I have up to 20 copies of each folder at the root level as well as multiple folder duplicates inside. I'll post this on the bug tracker. Anyone else tried this yet?

Are you sure you published the configuration (and not just start the storage pool)?
Screenshot?

Hi, i had the same Problem first,

after a full uninstall (clicking uninstall at the storage pool driver) and deleting the flexraid folder, the problem was gone and everything was published like in the preview.

Hey does anyone know with the automerge modes do you still need to have the complete folder structure on 1 drive for it to work?

Can you have multiple virtual drives? Every time I try to add the second virtual drive in similar manner to the first that works it refuses to map it.

No you can only have 1 storage pool.

webs0r wrote:Hey does anyone know with the automerge modes do you still need to have the complete folder structure on 1 drive for it to work?

To answer my own question - no you don't.
I've just moved to an automerge mode and ran a script to clean up all my empty folders on the previous "structure" drive.

I'm a long time user of your program and think that the effort and time you've put into this is tremendous. Thank you for this...

I've been using this as a backup option instead of WHS mirroring option. I've mostly used it to restore files deleted by mistake as I haven't had any hard drives fail on me yet (knock on wood). There are a few minor things I don't like about WHS, like it moving files around all the time and I've started to get errors concerning the tombstone files and other items. So, I've been playing around with the FlexRAID-view and the new GUI, planning on moving to a different OS.

I'm curious if there are any other differences between these modes. I really like the automerge options as I don't really care where the files are stored on the hard drives, as long as it's reasonable organized, where the "Auto-Minimized-Folder-Split-Priority" would fit me perfectly.

I've tested this setup on a 10 disk system and ran into something that doesn't occur using WHS. I have the hard drive information fairly organized, but thought that having all 10 drives combined would make it much easier to organize the rest of the hard drives into the right folders and locations. As I created some new folders and moved things around it started to move information from drives onto the next drive in the list with open space, even though I just moved it between folders. I would have thought the information would have stayed on the same hard drive and just put it into the correct folder without having to move all the files to the new hard drive.

Is this something that will be different with using the explicit configuration feature with FlexRAID? From what I've been reading it seems that _flxr_ is used more in that configuration setup. Would that be more of an "index" that is used to point to the file locations where the "Auto-Minimized-Folder-Split-Priority" configuration option just uses the actual file locations on the hard drives?

I tried to find any information pertaining to this, but was unable to do so. It's quite hard and time consuming to search through all the forum topics that has accumulated over the years. I think that you guys are working on a WIKI or a manual and I think that's a great next step.

@olgemap
The FlexRAID storage pool engine does not use pointers (links and whatnots) to organize the data.
Paired with that, the Auto-Minimized-Folder-Split-Priority mode does what it says... it tries to minimize folder split... i.e., it tries to keep folders and their contents together as much as possible.

WHS uses links (tombstones) on the landing drive to organize things. Doing that has its pluses and minuses including having to re-installing WHS to rebuild the tombstones.

The Explicit Merge mode was designed to address data organization in a logical manner without altering the physical content, which is what you want if I read you correctly.
The catch is, that organization takes place in explicit configuration definitions as opposed to move and rename operations on the storage pool drive.
In other words, you have to define virtual views with what they could contain.
Makes sense?

Yeah, I think so. So, as long as I define virtual folders in the base of the structure and keep my data under those folders, it will organize the the information logically and not physically. Thanks for the clarification...

This message was edited 2 times. Last update was at 01/05/2011 01:18:05

Need some help understanding the operation of the automerge feature.

I am migrating from WHSv1 to Win Server 2008, and want to use FlexRAID to mimic the drive extender pool feature. I plan to have a very simple folder structure on every drive:
Music
Videos
...Movies
...TV Shows

The first guide I found showed the explicit merge. I created a very simple config:
-K:\
|-*C:\Drives\DISK1
|-*C:\Drives\DISK2

This caused me to have 2xMusic and 2xVideos folders in the virtual drive since they were on both drives.

When I followed the auto merge in the OP:
-K:\
|-*Auto-Minimized-Folder-Split-Priority

I get the structure I want. However, one thing I don't understand is will it balance folders and files that are already on the drives? The drive extender feature sometimes would cause files, e.g. the movie file and its nfo which should be in the same folder, to be split amongst different drives. I want to avoid this as match as possible.

If the automerge is constantly balancing the files, then what happens if the file is in use during the balance? I think WHSv1 had a corruption problem due to the "landing" feature they initially had in the drive extender.

Thanks.

not that i know how FlexRAID-View works, but from my brief experience with "Auto-Balanced-Space-Priority", FlexRAID doesn't auto balance data already on my drives

so it probably means it doesn't move/consolidate data already on your drives

ShadowVlican wrote:not that i know how FlexRAID-View works, but from my brief experience with "Auto-Balanced-Space-Priority", FlexRAID doesn't auto balance data already on my drives

so it probably means it doesn't move/consolidate data already on your drives

Thanks. I am playing around with it right now. There is definitely a learning curve with FlexRAID.

This message was edited 6 times. Last update was at 10/05/2011 12:29:03

Stuck On Storage Pool: This might be detailed, but I wanted to give a breakdown of how a newbie (enthusiast) went through using the program & my roadblock(s).

I THINK I'VE DONE MY PREPARING: I've done a ton of reading as of late. I've read most articles on the forums >2011. It appears most of the good forum docs are replicated in the Wiki. Hopefully these following questions aren't answered somewhere else. I also have been running a WHS for over 1.5 years and wanted to run something more expandable & safe since one of my WHS drives is failing in a bad way.

MY COMPUTER: I bought five 2 TB HD's (brand new) for the sole purpose of ensuring if two hard drives go bad, I'll have *safe* data. I installed Win2k8 R2 on an older Q9400 box that has 6 onboard SATA. There's an OS drive + those five 2TB HD's. I formatted each drive with 4k sectors (they are WD20EARS) with letters D, E, F, G, & H. I'm not sure if this made any difference, but I created the partition with 1907000 out of 1907600 MB. Figured if I added any future drives & they for some odd reason had a smaller recognized size, I'd be safe. However I read somewhere that FlexRAID doesn't care if it's formatted or something, so maybe pointless?

MY FlexRAID SETUP: During FlexRAID config, I set it up for 3 DRU & 2 PPU & using the Tx engine; Tx seemed like the most flexible choice. I dragged out D drive to put under DRU1, E drive under DRU2, F drive under DRU3, G drive under PPU1, & H drive under PPU2. I left everything in Run-Time Properties alone. Under Command Execution Center, I clicked on 'Create'. I then tried out Update, Quick Validate, & Verify. Everything seemed to be working.

It took me a bit, but I found 'settings' under the mock start menu. After finding this thread, I went to storage pool & entered the Auto Balanced Space Priority settings exactly how it's typed in here (& clicked save):
I wasn't sure what to expect next. Since it said 'virtual' drive, I assumed that V drive would just show up in My Computer or Disk Management. I noticed it said the service needed to be restarted for changes to take effect. For simplicity sake I rebooted the computer with no V drive showing up.

QUESTION #1: Am I supposed to enable something else? I assumed this Storage Pool option is supposed to give me the full 6 effective TB worth of pool I have (since I'm using 3 DRU's & 2 PPU's on 2 TB drives). I figured it would show up as a drive similar to virtual CD drives, etc.

Less Important Questions:
QUESTION #2: Anyone have a link on External Network Shares? Don't know what that option is and figured it might be important for sharing my volume across the network.

QUESTION #3: If the sole purpose is to just protect data, why would anyone care to keep folder structure in place? Why would anyone want explicit anything, besides the energy savings? Isn't the point 'set it and forget it' more or less? You just pop in ever month or so to ensure no errors show up in the log after setting up the schedule properly.

----------

I hope I put this out there in the appropriate format. I sincerely appreciate any input on how dumb I must be not 'getting' the next step in making that magic virtual drive appear.

This message was edited 2 times. Last update was at 10/05/2011 13:02:39

If this ^ is too much to read (wall o' text), let me know & I'll condense/truncate it. I'm basically looking to use these 5 fresh 2TB HD's to make one single drive letter array in a snapshot raid form. I thought the storage pool settings was the correct way to go, but after a reboot it's not showing up in 'my computer'.

If this isn't the correct thread, let me know that too. If I need to submit a file or other settings, I'm more than happy to do so.

**EDIT** I also noticed this thread shows storage pools being in what I assume to be an "add a new config" option. I only saw the Storage Pool option in the mock start menu.

Davenrothz wrote:However I read somewhere that FlexRAID doesn't care if it's formatted or something, so maybe pointless?

Yeah, that was unnecessary for FlexRAID.

1. You need to start the storage pool.

2. That feature is a workaround for users who don't see their mapped network drive in the file explorer when trying to configure UoRs.
Essentially, it enables users to use UNC paths directly. Note that there currently is a bug with that feature.

3. Purely if you have existing data and you want to organize it to your liking. With the explicit configuration, you can create virtual views where you organize some of your data without actually moving them.
It's a powerful feature, you have to play with it to understand. The auto modes assume a certain convention. Each mode has its pluses and minuses, and it is up to you to determine what suits you best.

This message was edited 2 times. Last update was at 10/05/2011 16:38:38

Brahim wrote:1. You need to start the storage pool.

I believe I missed a step or there is something just not showing. I do not see anything that would allow me to start the pool.

Unlike that other screenshot, I do not have an option for a "Storage Pool Merge Configuration" inside of my "Configuration". However, his says "Flex View" instead of my simple name of "Dave's Media".