Messages posted by: NLS

Here...

http://www.openegg.org/forums/posts/list/176.page

I really REALLY wish you go though with this.

I wish it first of all selfishly: Because I want to see it working and use it.

But I also wish it because in the World of Software today (and I've been there, done that), things start like fireworks and then die like fireworks.
Not all, but the rate is increasing alarmingly.

In the end (if I wanted to suck you, I'd put this first), I wish because I'd like to see a smart project and a smart programmer recognized.

Remember that this path will have many rough turns. I am not talking about the code only. The hardest thing to manage are your users. This is almost the same for free, open source or commercial software. People (me included) subconsciously believe the dev is there for them and them only.

Anyway, I hope 2009 (HINT: 2009) will bring nice things in this community.

autega wrote:At the moment I am using Unraid on Ubuntu Server with a custom Kernel

!?!?

I think Brahim will just use a flexible RAID4 (like now but live, like "some other solution we know"). This would be good enough.
Now if he has a VERY smart idea that the big names in storage haven't thought, I am eager to see it.

I am sorry you believe that.

Your posts helped me, I have to say.

1) I learned how WHS deals with drive removal (will also verify on my virtual machine).

2) I am rethinking about using the single 1.5TB as parity and leave my two 750GB as data, even though I am actually "loosing" (not using) 750GB on my parity drive. At least this will allow me to add the second 1.5TB (when I will eventually buy it) with less drive juggling.

3) I am not rethinking about not using one of my four small IDE for my system drive. You see those four will maybe outlive even my largest SATA drives (as I can't replace those four 320G IDE with anything larger anyway... all future replacing will happen on my SATA buses). Maybe the only time I'd remove them would be if we have cheap 5TB SATA drives (each) and really those four IDE drives would then just eat power and bay space. Now I get a worth mentioning extra 1TB+ from those four old guys.

Again I am sorry if you think you lost your time.

??????
(testing)

...btw since you'll go into Forum settings, check if you can allow separate time-zone settings for every user (like all forums). It is very confusing reading posts with your own time stamp in your own time zone...

Hope so.

Thanks.

BTW, I understand your thinking about empty folders, but maybe you'd need to tweak things a bit and make FlexRAID recognize empty folders (even if it's just storing extra metadata). You see there are scenarios where certain setups need to have specific folders even if they are empty (not always empty, but could be empty at the time of rsync).

Rethink this please.

dscline wrote:

NLS wrote:So, maybe it would be better to use one of the four 320GB IDE for the system? At least this way the 750 data disk will utilize fully the 750 parity disk. What do you people think?

But what does that gain you? Your parity needs to be at least as large as your largest risk (largest data disk), but you don't really gain anything by it's space being fully or partially utilized. No matter how you configure your drives, you still need a 750G for parity. And no matter how you configure your drives, your total potential storage space is the sum of all your drives, less 20G for the system partition, less the parity drive. So it's the same either way.

No it's not. If I put my system on my larger data disk (one of the two 750G), then my larger data drive is 730G (as 20G are a different partition used for the system and the system doesn't go into FlexRAID), so I really need 730G parity, so I waste 20GB on my parity disk. I can use those 20G as "non-secured" storage (AND OUTSIDE WHS pool), which doesn't really interest me.
If I use one of my small disks (but one of those that are not going to be replaced any time soon - i.e. my IDE disks), then my larger data disk remains a full 750G and I fully utilize my parity.
The "waste" between the two options is that in the first I waste 20G of my parity. Yes the data space remains the same either way.

dscline wrote:As for your second scenario, how you do it is going to be a matter of personal preference. But for ME, my intent is to always keep my parity drive at least equal to my largest data drive (in other words, I don't plan to span my parity across drives, if for nothing else, simplicity, if not robustness). In your situation of maxed out drive connections, I would invest in a simple, cheap, USB drive cradle. When it's time to upgrade, you will likely be upgrading to larger drives, so get two. If you get two 1.5TB drives, put one in the USB drive cradle, and re-assign your parity to it (and build parity data on it). Then, take the drive that WAS your parity drive, and move it into the data pool. Then take one of the drives you want to remove out of the pool. WHS will then move all the data off of it, onto whatever space it will find (most likely, the newly available space that used to be the parity drive). Once that has completed, remove that drive and add in the other 1.5TB drive, and add it to the pool. Now repeat the process of taking another small SATA drive out of the pool, so that you can free up a connection to move the parity drive back into the case.

Well the point is that I don't want (or cannot afford) to buy TWO 1.5TB from the start. I can (maybe) buy one. Even if I buy none, I have to be "set" for the future. So you understand my scenario.

You helped me by saying what I have in bold above. You know for a fact that if I take a drive OFF WHS pool, then the data that was there automaticaly move to the other drives in the pool? How does this work? I first "unassign" the drive from the pool, let some process move the data and THEN physically remove the drive? Can you verify what I am saying?

dscline wrote:The next time you upgrade, if it's another 1.5TB drive, then it's just a matter of putting a 1.5TB drive in the external cradle, adding it to the pool, removing a small existing drive from the pool, then moving the new data drive into the case. Or if you are again bumping up in drive size, buy two and repeat the first procedure.

That at least is my plan. I built my WHS with three 1.5TB drives (plus the left over space from the system drive). One of the 1.5TB drives is earmarked for parity (I've installed WHS, but haven't loaded FlexRAID yet). I'm hoping that space will hold me until the $$/TB comes down on 2TB drives. At that point, I'll buy two, one of which will be the new parity drive.

My point is start by buying none or just one 1.5TB disk right now. So please re-read my original post with this in mind.

(also waiting for Brahim's comments)

Maybe you can arrange for that with my disks too.
I really suck in selling older equipment (or don't have the time - read it as you like).

Seems that the forum cannot accept non-latin characters.

Here are some Greek: ????????!

This is the first time I see this, any other forum I've used (and I don't mean Greek forums) works ok.

?????? ????....

I am getting very strange results!
(except if I understand something wrong in FlexRAID)

Before we go further:

- Is it normal that files that are ADDED after an rsync to the RAID, do not cause validate/verify to fail? (because it just checks for files it knows)

- Is it normal that empty folders are NOT included? (if this is normal, I am not sure it is good!)

If both the above are normal, then nothing is wrong in my log.log.

In fact in my wordpad the log entries that refer to the Greek filenames show normal:

[2009-10-02 03:58:55,455] DEBUG: Valid: C:\fs\I\DE\shares\Public\??????????????.txt
[2009-10-02 03:58:55,455] DEBUG: Valid: C:\fs\I\DE\shares\Public\?????????????????????????.txt
[2009-10-02 03:58:55,455] DEBUG: Valid: C:\fs\I\DE\shares\Public\????.txt

(I don't see any entry for the fourth entry that is the empty folder - in Greek - but this could be normal with FlexRAID)

So... Any other ideas?

At least it is good that it is only a display issue (both in webUI and command line)...

I haven't used FlexRAID-View... I don't know if we'd have an issue there.

EDIT: Grrrrrrr I see your forum software replaced my Greek with ?????... anyway they were looking ok before (and in the log).

OK thanks, will get back with this later.

So in practive, leaving let say just 100MB on my largest (same size as parity) disk, will be enough for my parity not to overflow...

I'd like to point WHS users (esp. experienced ones) to this post:
http://www.openegg.org/forums/posts/list/184.page
help please...

Since nobody else stepped in, let me (another newbie) tell you what I would do:

- leave C: out (you cannot RAID your system with FlexRAID yet)
- leave D: out (duh!)
- E:\ will be your PPU (btw, you'll have much empty space since you have very large parity drive... but *)
- N:\ will be DRU1
- O:\ will be DRU2
- P:\ will be DRU3
- Q:\ will be DRU4

...don't forget to "create" after that and don't forget to schedule rsync every time you think it's ok.

Simple.

(* you can use the empty space to store non-secured data, or you could just make this disk as two partitions and use a 400GB partition for your system -Ghost drive C- and then use your old C drive as data too: R:\ DRU5)

Brahim can you make a real estimation of the space parity + metadata needs?

Let me rephrase.

If I had a normal disk based RAID, I know that for all the bytes on a 750GB disk, I "eat" the exact same amount of bytes on my parity.

Since your system is on top of the filesystem...

Making parity for let say an exactly 750GB folder (a full disk with 0 bytes empty), would need HOW large a parity disk?
We have to remember:
1) on the same disk the metadata will be stored
2) the "real" 750GB parity file will in fact be split into many separate smaller files, each with each own header, directory entry etc.

...so in fact (thinking of those two above) eating more than 750GB total.

How much more?

Let me rephrase it further...

What is your "official" suggestion: How much empty space should I leave on my largest data disk, so that my same size parity disk won't be overflown?

(I'd love to hear something much smaller than 2GB)

I am still fighting the pros and cons of building my "FlexWHS".

Help me picture the following scenarios.

1st (easy) problem. This is a question for WHS users.
My current disk setup, consists of 2x 750 disks (one is for parity), 2x 500 disks, 6x 320 disks (THAT FOUR OF THEM ARE IDE).
My tower fits two more disks, but I don't have an empty SATA controller for them.
Even if I buy an extra controller, since I have space for two disks, it seems that the 4 IDE 320 disks, won't be removed for a long time.
Now my question is this: Generally people suggest putting WHS system on the larger disk, since this is usually the last disk you will upgrade. This has one problem in our case (let say I will not buy a larger disk at all): WHS needs 20GB, so I'll have an 750GB parity disk, for the largest "risk unit" being 730GB... 20GB empty on my parity disk. On the other hand, I probably will not be eliminating my four IDE disks any time soon and they will not get replaced by bigger either (as the largest IDE I can find is 500GB and too expensive - better buy 1.5TB SATA).
So, maybe it would be better to use one of the four 320GB IDE for the system? At least this way the 750 data disk will utilize fully the 750 parity disk. What do you people think?

2nd (more complex) problem. This goes more to Brahim. (btw this second problem probably gives one more vote to using one of the IDE 320GB disks for the system, see below for the answer to why)

So my initial scenario will be this:
SATA 750 - Parity
SATA 750 - Data
SATA 500 - Data
SATA 500 - Data
SATA 320 - Data
SATA 320 - Data
IDE 320 - Data
IDE 320 - Data
IDE 320 - Data
IDE 320 - Data + WHS

Let say I work fine with this setup for a month, then I go and buy a new 1.5TB disk. Remember I cannot add EXTRA disks (if I don't buy SATA controller) so I will need to replace one of my SATA 320 with this.

So, with the above scenario, WHAT exactly I need to do?
I'll be loosing 2 disks in fact, as the second 750 (that "now" has data), will need to become my second parity to fully secure my 1.5TB disk, plus I will have to remove a 320GB (IDE) disk that also already has data!
With those in mind, can Brahim describe the migration process from FlexRAID's point of view?
(well also from WHS point of view, because I wonder how I will "empty" the two disks that go off my data space... as I said the 320 will be removed from the system and the second 750 will stop having data and become parity)

(interesting trivia... buying this one 1.5TB disk will in fact give me only 400GB extra space )

Want to make it more difficult?
After a couple of months, let say I decide to buy one more 1.5TB disk, so I will need to replace one more of my 320GB SATA (remember there is no point replacing any of the 4 IDE disks, as I don't have empty SATA controller) with this new 1.5TB.
This will mean that: Possibly I keep the two 750 disks as parity (but for how long?) and just use the new 1.5TB in place of the 320GB.
OR (which is more complex but looks further into the future) I will need to move my two 750 disks BACK to my data pool and take the new 1.5TB as parity (remember there is already another 1.5TB disk in my data pool from the first step).
What are the moves I need to make?

Intriguing?

what do you mean?

I trust you are looking to solve this eh?
(eventually)