We have moved permanently! Join us @ http://forum.flexraid.com
We have moved permanently! Join us @ http://forum.flexraid.com
We have moved permanently! Join us @ http://forum.flexraid.com
[Logo] (Closed - visit http://forum.flexraid.com)
  [Search] Search   [Recent Topics] Recent Topics   [Hottest Topics] Hottest Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
Dillema: How / When to run Validate and Verify  XML
Forum Index » General discussion
Author Message
dscline


Joined: 23/03/2009 14:39:38
Messages: 233
Offline

I have several things scheduled to happen nightly on my server, while we are all likely asleep. One of those things is my nightly rsynch, which can run several hours if I've done a lot of ripping that day. There are a couple of other things that I have scheduled nightly, so the night time hours are pretty much consumed with various maintenance tasks. The problem with validate and verify is, they take SO long on a large array. So the result is: I never run them, because if I do, chances are someone will want to watch a movie after I've started it. The only solution I can think of would be to schedule it in small chunks using validateSet, but even with that, if I spread all the data out between the seven days of the week, I think that would still be several hours of validating per day, so I think if anyone happened to try to stream something from a drive that was currently being validated, it could cause problems.

Unless anyone has any other suggestions, I think that is probably the short term solution. But long term, would there be any possibility of having a way to do a low-priority task that could pause it's operation as soon as any other process started accessing the drive? I have a couple of other applications that do this: Stablebit Scanner, which monitors and tests drives for drive health, as well as my defragging tool. The way Stablebit works would be ideal: you tell it how often you want to check the drives (say, once every 30 days), you can optionally schedule it so it only works during certain hours, and you can optionally tell it to pause if other processes access the drive (with three levels of sensitivity). So if a drive is due for a check, it might start scanning it, but if something else starts accessing that drive, it will pause the scan and move on to checking an idle drive that is due for scanning. It works out well, because there's plenty of time when the server is sitting there idle, but outside of the middle of the night, it's hard to predict when that will be.

Perhaps this is a little too pie-in-the-sky, but I thought I'd throw it out there. I expect there are others like me who use their servers to stream HD media, but it's easy to get stuttering if you try to do something that is i/o intensive while you're streaming. We are all here because we want to protect our data, but I think validating and verifying are important too, not only to make sure our "back-up" is good, but also to make sure that bit-rot and other errors haven't crept in. But once your data-set gets large, these end up being very expensive tasks. Some way to run them during idle time would be a huge benefit, I think.

WHS v1
FlexRAID 2.0b10 T2+ 19 DRUs 2PPUs
Supermicro C2SEA, E8400 (stock), 4GB
Supermicro AOC-SASLP-MV8
Sans Digital 4ESPCIE (SiI3124)
Generic SiI3132
Brahim


Joined: 09/04/2008 23:28:33
Messages: 2883
Offline

What you want is the FlexRAID Expression Language feature.

Not only you will be able to combine tasks into workflows, but you will also be able to script lowering FlexRAID's CPU priority at the OS level.
Changing the CPU priority at the thread/task level does not work as expected.
Monitoring other processes for disk and CPU access and throttling based on that is possible, but work I will have to delay that for some other time.

Server (VMware ESXi): dual Quad 8356@2.4Ghz | ASUS KFN5-D SLI | 16GB (4x 4GB) DDR2 667Mhz ECC REG w/Parity [Chipkill] | Radeon X300 | Intel 160GB SSD (VM datastore) | 6+ TB storage
File Server VM (running FlexRAID): 512MB RAM | 2 vCPUs | 6TB storage | Parity on 2TB NAS
dscline


Joined: 23/03/2009 14:39:38
Messages: 233
Offline

Sounds like you're on top of it.

In the mean time, a couple of questions: If I try to schedule some small validates in chunks, and I try to fit it into my existing night-time processes, what happens if I have an rsynch that runs longer than expected, and then a scheduled validate (via command-line client) trys to start while the rsynch is still in process? Does the validate just not happen? Does it get put in some kind of queue and start when the existing task finishes? Or does it (gasp) cause the currently running task to abort?

Also, when it comes to breaking the validate up into chunks, I know there is the validateSet command that is based on files. Is there a validate version of VerifyStart and VeryifyNumOfBytes? It would be nice to break the chunks down by size rather than trying break it down into even chunks based on file paths.

Thanks!

WHS v1
FlexRAID 2.0b10 T2+ 19 DRUs 2PPUs
Supermicro C2SEA, E8400 (stock), 4GB
Supermicro AOC-SASLP-MV8
Sans Digital 4ESPCIE (SiI3124)
Generic SiI3132
Brahim


Joined: 09/04/2008 23:28:33
Messages: 2883
Offline

dscline wrote:Sounds like you're on top of it.

In the mean time, a couple of questions: If I try to schedule some small validates in chunks, and I try to fit it into my existing night-time processes, what happens if I have an rsynch that runs longer than expected, and then a scheduled validate (via command-line client) trys to start while the rsynch is still in process? Does the validate just not happen? Does it get put in some kind of queue and start when the existing task finishes? Or does it (gasp) cause the currently running task to abort?
...


Right now, if a task is running and another is executed, the later will fail.
The Expression Language is the solution to that issue by being able to define workflows.


...
Also, when it comes to breaking the validate up into chunks, I know there is the validateSet command that is based on files. Is there a validate version of VerifyStart and VeryifyNumOfBytes? It would be nice to break the chunks down by size rather than trying break it down into even chunks based on file paths.

There is not such thing for validate (unfortunately). Somehow, you will have to have the full list of your files and validate in batches.

Server (VMware ESXi): dual Quad 8356@2.4Ghz | ASUS KFN5-D SLI | 16GB (4x 4GB) DDR2 667Mhz ECC REG w/Parity [Chipkill] | Radeon X300 | Intel 160GB SSD (VM datastore) | 6+ TB storage
File Server VM (running FlexRAID): 512MB RAM | 2 vCPUs | 6TB storage | Parity on 2TB NAS
 
Forum Index » General discussion
Go to:   
Powered by JForum 2.1.8 © JForum Team



Locations of visitors to this page