[extension/filestorage] - Add an option to recover from panic if underlying file is corrupted.

[VihasMakwana]

Description

Add an option to recreate the database from scratch. This is useful if the underlying database is corrupted and there's no way to recover from it.

Ideally, we would want to recover from a panic. However, I came across the panic statements in bbolt's codebase and they are raised inside of a goroutine. Unfortunately, we can't recover from that panic in our collector process. So, our best route here is to provide an option to the user to recreate a new database.

Link to tracking issue

Fixes #36840, #35899

Testing

Documentation

[swiatekm]

The change looks good to me technically, but I'm not sure it's the right thing to do. My impression was that we wanted to recover from a panic when opening the DB, and then create the backup. Is that not feasible.

[VihasMakwana]

The change looks good to me technically, but I'm not sure it's the right thing to do. My impression was that we wanted to recover from a panic when opening the DB, and then create the backup. Is that not feasible.

Those were my intentions as well. However, I came across the panic statements in bbolt's codebase and they are raised inside of a goroutine. Unfortunately, we can't recover from that panic in our collector process.

Essentially, it is like following piece of code:

func main() {
	defer func() {
		err := recover() // WON'T HELP BECAUE OF PANIC INSIDE OF A GOROUTINE
		fmt.Println(err)
	}()
	go func() {
		panic("OOPS!")
	}()
}

[VihasMakwana]

@open-telemetry/collector-contrib-approvers could someone take a look here? I have an approval from the codeowner.