DFS alternative - Setup a JBOD like view

This message was edited 2 times. Last update was at 12/07/2008 15:45:19

Prelector's post on using NTFSLink got me to look more into that sort of tools as far as being paired with FlexRAID.

There are many alternatives out there that can achieve what DFS does to some extent.
One of the simplest alternatives is using the linking feature of the NTFS filesystem.

The best tool out there for NTFS links is: Link Shell Extension (LSE)

More info at: http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html

I will try to setup a video tutorial on this great tool some time later.

I'm not sure if I understand how this can be used to take the place of DFS. I use WHS which comes with DFS functionality. I haven't used any other of the DFS tools/utilities out there, but I would assume they behave in the same way.

DFS would create a data "pool" where multiple locations are combined into one folder/directory view. So, a DVD folder on disk1, 2, and 3, could be combined into one folder called DVD that is dynamically updated with the changes from disk1-3.

The issue to me seems to be that each link has to manually be created to the files/directories, just like a .lnk in windows. It does have additional benefits, but I'm not sure how it would be implemented as DFS.

If you could create a tutorial that demonstrates it, that would be great...

Thanks for your hard work. We all really appreciate the time and effort you are putting into this…

Yeah, I will create a video tutorial to showcase what the tool does.

You can use the tool to mimic exactly what DFS does.
In some regards, this tool can be viewed as more powerful than DFS since DFS only works on network shares.

This tool will mount both local drives/folders/files and network shares/folders/files.

I'm curious about this. Will the view using LSE look like one folder spanning multiple disks or will it be a folder with multiple subfolders underneath it?

Thanks..

-- John

brzez wrote:I'm curious about this. Will the view using LSE look like one folder spanning multiple disks or will it be a folder with multiple subfolders underneath it?

Thanks..

-- John

Well both.

You will end up with a folder whose sub-folders could potentially reside on other drives or even network shares.
Such folder will in that sense span multiple drives and have multiple folders (some of which will be mount points/links).

What if I want to have a single folder than can span multiple drives. I understand that WHS can achieve this by spreading data and maintaining links. I can also create a spanned volume in Server 2003, but then I loose the ability to remove a drive (break the volume) although I can still add to it. Is this why OpenAFS is required?

This message was edited 4 times. Last update was at 26/07/2008 10:18:53

FlexRex wrote:What if I want to have a single folder than can span multiple drives. I understand that WHS can achieve this by spreading data and maintaining links. I can also create a spanned volume in Server 2003, but then I loose the ability to remove a drive (break the volume) although I can still add to it. Is this why OpenAFS is required?

FlexRAID Basic does not have any data spreading feature.
For such thing to be possible, the application would need to fully control your system, which is not a design of FlexRAID Basic.
You will have to wait for FlexRAID NAS for such a feature.

I have promises a task that, in FlexRAID Basic, will automate the creating and maintenance of links for those using DFS, OpenAFS, and LSE.
I just have not had time to get to do it.

Also, I think that learning to leverage DFS, OpenAFS, or LSE is far more powerful than spreading data around.

In my configuration, my DFS view represents my storage cluster where the data reside not only on multiple disks but also on multiple computers in my network.
Also regardless of which computer I use, I have access to that consolidated view of my data.
In the screenshot below, the Z:\ drive points to data that reside across my network.

At the end of the day, you have to organize your data and determine where it should best reside.
So, the data spreading feature, though I might add it, is not something that really excites me. But of course, it is not all about me.

So then, in your above example (pic), there is no way to consolidate DVD1, 2, 3, 4, 5 into a single 'space'? So if you are trying to access a file and you need to search through five directories to find it, and if you are going to add to this DVD folder, you need to decide which of the five subfolders to add too. Is this right?

FlexRex wrote:So then, in your above example (pic), there is no way to consolidate DVD1, 2, 3, 4, 5 into a single 'space'? So if you are trying to access a file and you need to search through five directories to find it, and if you are going to add to this DVD folder, you need to decide which of the five subfolders to add too. Is this right?

There is a way to consolidate DVD1, 2, 3, 4, 5.
The catch is, it is tedious and you will need a script to make it worthwhile.

What do you mean by "if you are going to add to this DVD folder, you need to decide which of the five subfolders to add too. Is this right?"?

To the RAID?
In my setup, DVD1, 2, 3, 4, 5 are on separate disk and hence will be added to their own DRU.

That said, I RAID my data outside of DFS.
The RAID is on the server and is configured using the local disk paths.
This screenshot view is a view from my workstation.
The workstation only has network access to the DVDs.

Unless your intent is to create a RAID that includes network data, you don't want to use the DFS view as the basis for the RAID as it abstract where the data actually reside.

Again, drive Z:\ represents my consolidate network data.
I can write to Z:\ without caring where it is being written to unless there is a space or security issue.
Z:\ also looks the same regardless of which computer I use.

There is a way to consolidate DVD1, 2, 3, 4, 5.
The catch is, it is tedious and you will need a script to make it worthwhile.

That doesn't sound very tempting. I don't want anything tedious or that requires much maintenance.

What do you mean by "if you are going to add to this DVD folder, you need to decide which of the five subfolders to add too. Is this right?"? To the RAID?

No, to the Z:\DVD\ folder shown in the pic, it has 5 subdirectories listed. So if you want to add files to the DVD directory, you need to choose where to put the files. 'This one goes in DVD1, but I'll but these ones over in DVD4.' Or, when you access the Z:\DVD\ does it just appear as a single folder?

In my setup, DVD1, 2, 3, 4, 5 are on separate disk and hence will be added to their own DRU.

OK, I don't follow this. If they are all on a separate disk (not disks), then why add them this way? Why not just have one (DVD1)?

This screenshot view is a view from my workstation. The workstation only has network access to the DVDs. Unless your intent is to create a RAID that includes network data, you don't want to use the DFS view as the basis for the RAID as it abstract where the data actually reside.

No, my intent is to create a large secure file storage, across many drives, but on one server ... but not to be restricted to the requirements of typical RAID. Essentially what unRAID provides, but on a Windows server.

That doesn't sound very tempting. I don't want anything tedious or that requires much maintenance.

It is tedious if you do it manually.
The script will automate things and will run at set schedules.
With all solutions, there are compromises.
Again only FlexRAID NAS will provide something that is totally transparent.

No, to the Z:\DVD\ folder shown in the pic, it has 5 subdirectories listed. So if you want to add files to the DVD directory, you need to choose where to put the files. 'This one goes in DVD1, but I'll but these ones over in DVD4.' Or, when you access the Z:\DVD\ does it just appear as a single folder?

Yeah, it is not the most ideal.
However, you will find that most of the time it does not matter.
I use MyMovies to catalog the movies. So, where the actual files reside becomes moot.
Most of your content that will span across multiple disks would need some sort of management tool anyway.

OK, I don't follow this. If they are all on a separate disk (not disks), then why add them this way? Why not just have one (DVD1)?

The "s" is missing.
There are on separate disks.

No, my intent is to create a large secure file storage, across many drives, but on one server ... but not to be restricted to the requirements of typical RAID. Essentially what unRAID provides, but on a Windows server.

Then you will need something that controls your reads/writes like WHS, unRAID, or in the future FlexRAID NAS.

That clears everything up, thanks for all the info!

Looks like I'll be cracking open an ice-cold WHS to see what it's like. Hopefully it's not that annoying.

Cheers, eh?

My question is ... how do you track disk space?

Say I have 5 drives on a box .. each it's own volume. I create a link for each drive into my DFS Root folder.

Now I mount that root folder on a different machine on the network. When I open it, I see 5 folders inside (one for each physical drive).

But the actual mounted folder reports the size of the drive that the root folder lives on. I have no way of knowing (other than looking at the actual host machine) how much disk space is available.

Is this correct or am I missing something?

rhubarb9999 wrote:My question is ... how do you track disk space?

Say I have 5 drives on a box .. each it's own volume. I create a link for each drive into my DFS Root folder.

Now I mount that root folder on a different machine on the network. When I open it, I see 5 folders inside (one for each physical drive).

But the actual mounted folder reports the size of the drive that the root folder lives on. I have no way of knowing (other than looking at the actual host machine) how much disk space is available.

Is this correct or am I missing something?

Yeah, unfortunately that's correct.

I can make it work

BTW .. I just dumped DFS in favor of Link Shell Extension (LSE) ... works GREAT!

I'm curious on how you "made it work".

Thanks..

Brahim wrote:

FlexRex wrote:So then, in your above example (pic), there is no way to consolidate DVD1, 2, 3, 4, 5 into a single 'space'? So if you are trying to access a file and you need to search through five directories to find it, and if you are going to add to this DVD folder, you need to decide which of the five subfolders to add too. Is this right?

There is a way to consolidate DVD1, 2, 3, 4, 5.
The catch is, it is tedious and you will need a script to make it worthwhile.

Here is a VERY helpful script posted to AVSforum Flexraid thread. havent had a chance to test it but seems to do exactly what i want

Hi,

I'm not actually using FlexRAID yet but I intend to. I like the idea of it very much. In the meantime I've been playing with DFS to get it working how I want it. I've written a script to trawl through my folder structure and add everyting into the DFS Root so that I see a consolidated view of my files.

My folder structure is something like below:

Disk 1

Movies001

Star Wars

Bladerunner

Disk 2

Movies002

Lord of the Rings

Shrek

Disk 3

Movies003

Jarhead

That's just an example, but you get the idea. Each of the Movies folder is shared as a hidden share. Movies001 is shared as movies001$, Movies002 is shared as movies002$ etc.

I then have a DFS Root called Movies. This is just placed on the system drive as it's only full of pointers. The script goes through each of the Movies shares and adds each subfolder as a node to the DFS root. Here's the script:

Option Explicit

Dim colShares, objShare
Dim objFolder, colSubfolders, objSubfolder
Dim strComputer : strComputer = "."
Dim objWMIService : Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Dim objDfsNode : Set objDfsNode = objWMIService.Get("Win32_DfsNode")
Dim objFSO : Set objFSO = CreateObject("Scripting.FileSystemObject")
Dim errReturn

' Set these variables
Dim strShare : strShare = "movies"
Dim strServer : strServer = "server01"
Dim strDFSShare : strDFSShare = "movies"

' Retrieve collection of hidden user shares of the Disk Drive type
Set colShares = objWMIService.ExecQuery("SELECT Name, Path FROM Win32_Share WHERE Name LIKE '" & strShare & "%$'")

For each objShare in colShares
Set objFolder = objFSO.GetFolder(objShare.Path)
Set colSubfolders = objFolder.Subfolders
For Each objSubfolder in colSubfolders
errReturn = objDfsNode.Create("\\" & strServer & "\" & strDFSShare & "\" & objSubfolder.Name, "\\" & strServer, objShare.Name & "\" & objSubfolder.Name, objSubfolder.Name)
Next
Next

I actually have separate scripts for each of my share types; movies, episoded, music etc. To make things easier for this post though I've abstracted it out slightly to make the script work for anything. All you need to do is change the following variables to match your environment:

strShare
strServer
strDFSShare

Save the script as dfs.vbs for example and you can run it from the command line using the following command (run it locally from the server):

cscript dfs.vbs

This could of course be scheduled. When it's finished you'll have a share showing something like the following:

\\server01\movies

Star Wars

Bladerunner

Lord of the Rings

Shrek

Jarhead

A nice consolidated view. Hopefully this script will help someone. It works perfectly for my needs anyway.

Cheers,

Adam

This message was edited 2 times. Last update was at 07/08/2008 05:31:19

EDIT: LSE (link shell extension) can do what I propose underneath and even make junction safe on XP. Thats what you get for not reading the thread.

Forgive me if this has already been mentioned but, I believe there is a very and I mean very easy solution to having all your data in one virtual folder/directory on windows. The catch... needs vista or better. You can create symlinks in vista that are much better and more user friendly then junctions in xp or server2k3.

Why you should care...

Junction makes a hard link, which has a lot of dangers associated with it. eg. deleting a hard link deletes that data it was pointing to.

vista's symlinks are basically shortcuts. When you don't want one anymore you simply delete it. All your data still exists but the symlink doesn't anymore.

the follwoing is a screen print of the command mklink on vista

MKLINK [[/D] | [/H] | [/J]] Link Target

/D Creates a directory symbolic link. Default is a file
symbolic link.
/H Creates a hard link instead of a symbolic link.
/J Creates a Directory Junction.
Link specifies the new symbolic link name.
Target specifies the path (relative or absolute) that the new link
refers to.

this would enable you to have one master directory with symlinks to all of your content underneath it.

eg. mklink /d c:\stuff\hdd1 e:\
mklink /d c:\stuff\hdd2 f:\

I actually use this already, so I know it works.

So basically you create a real folder
then populate it with symlinks to the actual data

This message was edited 1 time. Last update was at 07/08/2008 18:42:14

This is a very good post as people need to understand the danger of junction links.
Everyone should really understand the differences, features, and limitations of the various linking tools out there.

goose wrote:EDIT: LSE (link shell extension) can do what I propose underneath and even make junction safe on XP. Thats what you get for not reading the thread.

Forgive me if this has already been mentioned but, I believe there is a very and I mean very easy solution to having all your data in one virtual folder/directory on windows. The catch... needs vista or better. You can create symlinks in vista that are much better and more user friendly then junctions in xp or server2k3.

Why you should care...

Junction makes a hard link, which has a lot of dangers associated with it. eg. deleting a hard link deletes that data it was pointing to.

vista's symlinks are basically shortcuts. When you don't want one anymore you simply delete it. All your data still exists but the symlink doesn't anymore.

the follwoing is a screen print of the command mklink on vista

MKLINK [[/D] | [/H] | [/J]] Link Target

/D Creates a directory symbolic link. Default is a file
symbolic link.
/H Creates a hard link instead of a symbolic link.
/J Creates a Directory Junction.
Link specifies the new symbolic link name.
Target specifies the path (relative or absolute) that the new link
refers to.

this would enable you to have one master directory with symlinks to all of your content underneath it.

eg. mklink /d c:\stuff\hdd1 e:\
mklink /d c:\stuff\hdd2 f:\

I actually use this already, so I know it works.

So basically you create a real folder
then populate it with symlinks to the actual data

This message was edited 1 time. Last update was at 10/01/2009 05:38:56

Hi,

is there a simple step by step on how can I use LSE to become DFS like

Thanks