Messages posted by: spiffyrex

In that case please don't let me distract you. I thought that FlexRaid Basic and Live! had completely different codebases since you referred to Basic as a proof of concept. But if adding features at this point will complicate your good work please don't worry about it. I really like FlexRaid Basic, so I can wait for it to become open source!

Is this fragmentation also an issue for the Linux version? My parity drive has an XFS filesystem and the fragmentation is less than 10%. However, I have only run rsynch a couple of times.

Hello Brahim, any news on making FlexRaid Basic open source? I would love to implement this feature myself.

I don't think retrying will do much good.
Once a sector is bad, it is bad.

The problem is that there is no industry standard as to what a bad sector is... For example, a drive targeted for workstations will retry several thousand times before marking the sector bad. It could take minutes, instead of a fraction of a second, to read just one sector. A drive targeted to the server market will just retry for a few seconds before giving up. That's because such drive will most likely be part of a hardware based RAID and the controller could mark the drive as bad if it takes too long to report a success or failure. In my case I just had to read the file a couple of times. Eventually the drive got a successful read, marked the block bad, and moved the data to another block in the free list.

Going from Windows to Linux, you would need to reformat your drives and use a Linux filesystem in order to use something like MHDDFS.
So, you will end up having to recreate the RAID.

No need to reformat, ntfs-3g is available in Ubuntu and other Linux distros. It allows you to mount an NTFS filesystem r/w. You can also create, check, change size, etc. I have been using Ubuntu 8.04 and formated new drives with XFS but kept old drives with NTFS. Not fun to backup 1TB drives...

Well, I vote for Linux + MHDDFS. My goal is to run Ubuntu + FlexRaid + funionfs from a 4GB USB flash drive. That way I don't have to install anything and can let friends borrow or make a copy so they can try FlexRaid in their system. Although possible, it's really difficult to do that with WHS...
Like mhddfs, funionfs will aggregate or join filesystems. I think mhddfs has more options, but funionfs has been available for a while so that's what I use for now.

One more HUGE bonus to me about WHS: all WHS backups can be mounted and opened, with full access to all of the files, by any WHS connected client PC. This allows you to view and restore individual files, in addition to entire volumes.

This is true of recent OS/hard-drive/partition backup solutions these days. Norton Ghost and partimage (from the SystemRescueCD freeware) do that. I do agree that WHS is easier to use but IMHO it's not worth the $$$. Also, I can't legally make copies and give them to my friends.

Great, tks. I take that increasing the the system memory to 4GB is not going to make it faster. At least for the FlexRAID server app.

I was running verify on a big array last week and I got a parity error. Sometimes hard drives get bad sectors but they can be relocated. So I did a copy of all the files involved to a temp directory (and then removed them) to allow the drive to relocate the bad sector(s) if possible. Looking at the S.M.A.R.T settings in all the drives, there was a new relocated block count in one of them. Then ran verify again and this time it completed successfully.
It looks like verify doesn't retry a parity error. Could it retry a few times, or even better, can there be a retry count variable where we can tell verify how many times to retry a calculation before failing it? It will be also good if it prints a warning every time there's a retry.

>1. How far in do you get the out of memory error?
Luckily just a few minutes. My guess is that it computes what needs to be done and then runs out of memory as soon as it starts calculating parity.

>2. How many files do you estimate in your RAID?
627766

>3. How much memory do you have in your system?
>Now, do me this favor: give me the "peak" memory usage for the host process as reported by task manager
Forgot to mention that I'm running Ubuntu 8.04. I have 2GB of memory, of which 1.82GB is available. The OS memory working set, without Firefox, FlexRAID GUI (Apache), or any other big app running is about .56GB. Once that I start a create or rsynch memory usage jumps to 1.72GB. I see it go as high as 1.8GB during a create or rsynch. There's plenty of paging space available and it looks like the server is not using any. Which is a good thing since there's no sense in generating more disk IO. One odd thing is that memory utilization stays at 1.7GB even after a create finishes and even after I shutdown the FlexRAID server app.

>I take it you get an "Invalid DRU..." error message. Right?
Right:

[2008-07-15 13:51:44,017] ERROR: [create] Invalid DRU format: DRU1{/flexraid/d1}|DRU2{/flexraid/d2}|DRU3{/flexraid/d3}|DRU4{/flexraid/d4}|DRU5{/flexraid/d5}|DRU6{/flexraid/d6}|DRU7{/flexraid/d7}|DRU8{/flexraid/d8}|DRU9{/flexraid/d9}|DRU10{/flexraid/d10} at: DRU10{/flexraid/d10}
java.lang.Exception: Invalid DRU format: DRU1{/flexraid/d1}|DRU2{/flexraid/d2}|DRU3{/flexraid/d3}|DRU4{/flexraid/d4}|DRU5{/flexraid/d5}|DRU6{/flexraid/d6}|DRU7{/flexraid/d7}|DRU8{/flexraid/d8}|DRU9{/flexraid/d9}|DRU10{/flexraid/d10} at: DRU10{/flexraid/d10}
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)

>FYI: avoid using DRU0 in the future.
Sure, I just make d0 part of DRU1. It would be nice if create fails or puts a warning when someone uses DRU0.

I'm having trouble running rsynch in a big array. I'm configuring a 5TB array which is 3.7TB full. I can create, validate, verify, and restore but rsynch fails. Everything works if the array is just a few GBs full. All DRUs and PPUs are local to the FlexRAID server.
This is the console output:

FlexRAID 1.0 RC3 build 006

Started: July 16, 2008 4:59:28 PM CDT

[2008-07-16 17:08:10,494] INFO : Total process size = 46424766424
Uncaught exception: Out of memory
Exception in thread "Thread-2" java.lang.OutOfMemoryError
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)

This is the log output:

[2008-07-16 16:59:28,936] DEBUG: Starting FlexRAID Host Service...
[2008-07-16 17:00:11,429] DEBUG: Executing rsynch...
[2008-07-16 17:08:10,494] INFO : Total process size = 46424766424

These are the relevant fields I changed in create.txt:
buffer=64KB
processes=2
leaveOff=30MB
metadata=/flexraid/p1
data=DRU0{/flexraid/d0}|DRU1{/flexraid/d1}|DRU2{/flexraid/d2}|DRU3{/flexraid/d3}|DRU4{/flexraid/d4}|DRU5{/flexraid/d5}|DRU6{/flexraid/d6}|DRU7{/flexraid/d7}|DRU8{/flexraid/d8}|DRU9{/flexraid/d9}
parity=PPU1{/flexraid/p1}

These are the relevant fields I changed in rsynch.txt:
buffer=64KB
processes=2
leaveOff=30MB
metadata=/flexraid/p1

This fails in RC3.003 and RC3.006.
Also, there seems to be a limit as to how many DRUs we can configure, So far DRU0 to DRU9 work, but create fails if I put DRU10. Not a big deal, 10 DRUs are enough for me. Oh, and all DRUs have to be populated by at least one file, but that's a known issue.

Thanks is advance and please keep the good work. I'm very impressed with how useful FlexRAID is!