First, I need to do some clean up on the naming of the products.

FlexRAID:
FlexRAID is a concept and not a product in itself. The core essence of that concept is flexibility and solving many of the shortcomings that plague current storage solutions.

RAID-F: Also known as RAID over Filesystem.
RAID-F provides data protection and data pooling over existing filesystems. It does that by overlaying its own lightweight and unifying filesystem on top of any filesystem that the user’s OS can operate over.

NZFS: The “Not ZFS” RAID system.
Although having no commonality with ZFS, NZFS borrows a number of concepts from ZFS and its RAID suite.
NZFS has two modes of operation: RAID under filesystem and RAID within filesystem.

In essence, FlexRAID will provide:
- RAID under filesystem (NZFS)
- RAID within filesystem (NZFS)
- RAID over filesystem (RAID-F)

Each approach has it own implications (i.e., advantages and disadvantages), which we will get into once NZFS is out.

On top of being very restricted resource wise, one must entirely obey these guidelines: http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q186775&
Not only bugs in kernel mode executions are unforgiving (blue screen of death anyone?), they are also non-obvious (get ready to call WinDbg your new best friend and time consuming’ly so).

Unless you are a hardcore and lifelong dedicated kernel mode developer, the constraints and learning curve are far too steep.
My only salvation in this whole initiative is that I have been dabbling in kernel mode development ever since I started FlexRAID. This has given me enough time to slowly get comfortable with this very niche type of programming.

In any case, I really wanted to minimize my kernel mode execution. The rewards were obvious: less pain and less bugs.
Moreover, most of my libraries are user mode libraries and porting them to the kernel means some serious refactoring.

I thought hard and long and fought myself between “easy street” and the “right way”.
Yet, as Spike Lee would say, “Do the Right Thing” nudged me along the side of the “right way”.
And so I did and ported everything into the kernel.

Obviously, there are many advantages to going 100% kernel mode, but I will leave those points for another post.
The key point in all this is that development of NZFS will take much longer given the now longer debugging and testing time it requires.

Nevertheless, I have made great progress as everything has been successfully ported to the kernel.
Simple NZFS volumes work nicely including RAID 0 and mirrored volumes. I will be testing parity based RAID volumes this weekend.
If all works out, the next step will be implementing the management and maintenance components.

For those who don’t, scrubbing is a full data scan to check on it integrity. Virtually every RAID implementation relies on scrubbing as part of its maintenance to ensure continuous data integrity.

In FlexRAID’s existing implementations, scrubbing is implemented through the Validate and Verify/Force-Sync-Verify tasks.
For NZFS, I really wanted to do without those long running tasks in order to achieve a zero maintenance design.

Well, I have an idea, but my lips are sealed on it.

With the recent announcement of Storage Spaces by MS, it was clear that FlexRAID needed to step up in order to stay ahead of everyone in the storage pooling space.

The current FlexRAID implementation of storage pooling has several key advantages over everything else on the market or planned including:

• Better power saving features (only the disk where the data resides needs to be active)
• Support for drives with existing data (FlexRAID never format any drive)
• A drive taken from a FlexRAID pool is fully readable outside of the pool and on any other computer
• Snapshot RAID when real-time parity synchronization is not necessary
• Real-time RAID
• Ability to restore specific files instead of the whole disk
• Support for network drives in the storage pool
• Disk spanning for better protection level and utilization
• Multiple RAID engines including support for RAID∞

That said, it has certain inherent limitations because it lives above the host filesystems.

NZFS takes a different approach and provides pooling below the filesystem. That’s right, a NZFS storage pool will need to be formatted with your favorite filesystem be it NTFS, FAT, EXT, etc.

I am calling it NZFS because it is not ZFS but will bring many of the ZFS features such as checksum, ZIL, de-dup, copy-on-write, etc. to both Windows and Linux.
Furthermore, it will be powered by the almighty FlexRAID’s RAID∞ engine.

Mirroring is crap and greatly inferior to multi-parity in all facets including protection level, space utilization, and ultimately cost.

Note that NZFS is not an improvement over the current storage pooling approach used by FlexRAID. It is just different and might be better suited for certain deployments.

More on all this later… busy coding.

Well, FlexRAID concerns itself with the data and not the storage. Hence it concerns itself with storage data (data that is stored) as opposed to transactional/transient data (data that is not stored).

Data storage management leads to managing the storage mediums, which is mostly a secondary concern of FlexRAID.
In fact, the sole form of storage management function in FlexRAID is provided thanks to its client UI as the core does not concern itself with such thing at all.
It is the UI that provides disk organization and S.M.A.R.T monitoring as a form of disk storage management.

In sum, FlexRAID focuses only on data and more specifically on storage data.

However, this will be discussed in a different post.
For now, here are a few UI spoilers.

- The shares defined in FlexRAID integrate into Windows Share Management interface for more fine tuned management.

- Through the Windows Share Management interface, you can define advanced settings including NTFS permissions.

- Although share permissions can be changed here, it is best to change them through the FlexRAID UI.

With that, FlexRAID now provides a similar and consistent user experience regardless of the platform.
Users can choose to run FlexRAID on Windows or on Linux based on their requirements and take advantage of what each platform has to offer.

1. Cruise Control Panel

2. Cruise Control Drive Manager

3. Storage Pool

It is important to write bug free and user proof code, but the reality is without full control of the user’s environment, that’s nearly impossible.
The next best thing, however, is the ability to auto-correct. Whether it be from system crash, user error, bug or whatnot, being able to recover and auto-fix is quintessential.

FlexRAID in Real-Time RAID mode can now self correct without needing to bring the storage pool down. That’s right; FlexRAID can now self correct live!

The next step will be to make the RAID self-heal the data itself from data rot stemming from memory errors or failing drive errors. The foundation for this next feature is already in the Snapshot RAID implementation. It is now just a matter of bridging the two to close the circle.

FlexRAID requires no plugin and presents its Storage Pool drive as a native NTFS drive to WHS 2011.
This means you can add and manage a FlexRAID pool drive just like a regular but massive drive in WHS 2011′s dashboard.

1. A clean and fresh installation of WHS 2011 + FlexRAID

2. Launch the FlexRAID Web UI

3. Create a new Cruise Control configuration and specify the type of RAID protection you desire

4. Open the configuration and go to the Drive Manager

5. Use the Drive Manager to manage your storage system

6. Add new drives to your Storage Pool

7. Add as many PPUs as you require tolerance for your RAID

8. Initialize your RAID

9. Start the Storage Pool

10. After the Storage Pool has started, inspect your newly created drive

11. Verify that WHS correctly picked up the FlexRAID Storage Pool drive

12. Create your server folders using the WHS 2011 dashboard

13. Confirmation of a successful server folder creation

14. Enjoy FlexRAID on WHS 2011