Idea for safer FlexRAID

Right now FlexRAID stores metadata separately, default on parity folder, but configurable anywhere.
I trust those metadata also hold the pool details (even if the pools are more than one).
If not, then I guess they are stored somewhere (like some system folder or FlexRAID install folder or you tell us where). I don't know.

This means that to be able to recover from a problem, you need the parity AND the metadata AND the configs (if the metadata do not carry the config details). What happens if the problem is exactly where the metadata and/or config files are? How does FlexRAID kickstart then?

Currently the user has to have a backup somewhere. A RAID system that asks you for a backup? Hm...

How should it be?

1) Default the metadata (and config files if needed) on a different folder from the parity (preferably in some system folder accessible by all users).

2) "Mirror" the metadata (and config files if needed) IN THE PARITY file(s).

What do you gain from that?

If you loose the parity folder (or disk if we talk about full disks), then the metadata (or config) needed to rebuild the array and it's parity, are somewhere else.

If you loose the disk (or folder) where the metadata (or config) are normally stored, then the parity file(s) can recover that too (also check my PM to see some extension to this).

In other words, you have two points of failure not only for the data, but also for the arrays themselves!

That's how it should be.

Hardware RAID solutions (talking about expensive server solutions not 50 euro home solutions), use (1) local flash memory and (2) the array disks themselves, to store the array configuration.
So whichever disk you need to replace, doesn't mean you lose your configuration (of course). On the other hand if you need to replace your RAID card (with a same model of course), you STILL don't lose your configuration because the new card looks in the disks connected on it to see if a configuration already exists. Finds it and uses it.

This is what I am trying to emulate with my idea.

What do you think?

(and easy to implement even for 1.3 final)

1. The metadata is all you need. It includes config information.

2. The default behavior is to save the metadata with the parity information.
So, the only way to lose the metadata would be to lose the parity information.

If you choose to move the metadata to another place other than with the parity data, then it is up to you to protect it.

Yes, that is clear.

I am proposing a better way.

Think about it.

Why should be up to the user to protect the system that... protects his system? It should protect itself.
The job of making sure a properly configured FlexRAID keeps working, is FlexRAID's job for itself.

Having (and supporting) a metadata duplicate, is simple and effective: Your system looks in the "normal" location and if it cannot find it, looks in the parity folder. If it finds it in either location, system makes sure to copy it back to the other location. This doesn't need to be scheduled, it can transparently be done before any rsync, verify etc. No bother to the user (except maybe some log entry). Job done, simple and effective and resulting in a safer array.

I know FlexRAID is still "incubating" so it doesn't have to be 100% friendly and feature rich, but I consider this basic functionality (and very easy for you to add).
Don't make the user do things that HAVE to be done anyway FOR the system (so why not BY the system).

In any case I propose an idea, which is not even mine. It's how stable systems work. I hope you consider it.

Backup to where? Another configuration option?
I see this as a valid but low priority feature request.

Again, the default setup is the recommended setup: leave the metadata with the parity data.
This is the best and most logical option.

If you have a need to move the metadata to somewhere else (advanced configuration), then it is up to you to figure out how to protect it.

I am not talking about any configuration option.
I am not talking about advanced configuration. It should be the default behavior.

Your logic covers for when the data folders/disks get messed, but doesn't cover for when parity folder/disk gets messed (along with metadata file). Parity disk/folder has exactly the same statistical possibility to get messed as the data it protects.

Metadata should be stored in two default places and FlexRAID should check that they are there (at both places). One place should be the parity folder (as is now) and the other somewhere else (possibly host installation folder) - hopefully a different disk.

That's all I am saying. I hope it's clear. My English sometimes can't get the message through.

Anyway.

I do get your point.
I think my argument isn't coming across as cleanly as I hope it would.

Your request is valid, but it is just that the concerns you have should not be so.

Taking the default configuration, if you lose the drive where the metadata is, you are also losing the parity data.
Just like regular data, you can restore the parity data by.... recreating it along with the metadata.

So, whether you lose your parity drive (and metadata) or lose a data drive, you need to do some recovery (or recreation).

Once you lose the parity data, the metadata itself is useless.

I am trying to learn about raid so aplogies for being a bit of a noob but I have a question about this discussion if its not too much to ask just to check I understood it to at least some degree

the parity and metadata although seperate entities are dependant

So metadata without parity or parity without metadata would mean the actual data itself would then be used to not just recreate the missing one (parity or metadata) but to create both?

So going back to the initial suggestion, rather than having an additional backup of metadata a better suggestion would have been to have a built in second copy of both metadata AND parity stored in another location on another drive

Which I guess would only be advantageous if it was noticeably quicker to just copy the "working" set of metadata and parity from a second location to the primary one rather than creating it from scratch from the data itself if totally lost

Kind of like storing your data in raid 5, and then having a raid1 mirror of your metadata and parity?

How far off am I with that interpretation?