Shutting off machine at the wrong time results DeltaChat being stuck in various conditions, most commonly loops of "Not moving the message XYZ that we have seen before."

[ell1e]

(based on issue deltachat/deltachat-desktop#4842 )

Sometimes, DeltaChat Desktop no longer syncs on one of my devices. This seems to be potentially a core bug. The UI sometimes will say "Updating..." for a while, but the messages shown in the main contact list never update and they're stuck roughly 10 days in the past, and no new messages are showing. Any individual chat won't load, it just say "Select a chat or create a new chat" and nothing ever happens.

Here's how the main contact list looks like while stuck:

During investigations, I obtained a --devmode terminal output log file which shows the following error: 7.2s �[1;33m[w]�[0m�[0;37mcore/event�[0m: 2 src/scheduler.rs:731: Failed fetch_idle: fetch_move_delete: fetch_new_messages: database disk image is malformed: Error code 11: The database disk image is malformed

• Operating System (Linux/Mac/Windows/iOS/Android): Linux
• Core Version: v1.156.2, v1.159.4
• Client Version: 1.54.2, 1.58.2

Expected behavior

Shutting down the device at an unfortunate time won't leave Delta Chat unable to sync after the device starts up again.

Actual behavior

Shutting down the device at an unfortunate time apparently will leave Delta Chat unable to sync after the device starts up again.

Steps to reproduce the problem

1. Use Delta Chat
2. Hard shut off your device with power loss at just the wrong moment
3. I use btrfs. This means any sort of checksum failure due to filesystem corruption should lead to an I/O error on open()/fopen() and not allow any reading in the file, and I think there should also be a dmesg message. Since that didn't happen, I don't think a filesystem corruption is the likely cause here (although I did have that on the PinePhone in the past, likely due to how unreliable SD cards are, so I thought I would point that out to be completely transparent).

Screenshots

see above

Logs

see above

[iequidoo]

This doesn't look as a Core bug, but rather as an SQLite bug (but unlikely) or some problem with sync requests implementation by your device, e.g. they don't work as write barriers (https://www.sqlite.org/howtocorrupt.html mentions this along with other possible reasons, you can check them). Btrfs mustn't necessarily detect such corruptions by checksumming because AFAIU checksums are calculated separately for allocated fs blocks, but if blocks are written in the wrong order by your device, that won't help. Which device do you use?

[ell1e]

The PinePhone uses an SD Card, so I assume "Disk drives that do not honor sync requests" is potentially true. But the contents shouldn't be corrupted since BTRFS would catch that (it did in the past, so that's not theoretical).

[iequidoo]

It's unlikely that Btrfs checksums can protect from reordered writes because they are calculated for blocks independently. You would rather get some fs metadata mismatch, but if just two data (not metadata) blocks are written in the wrong order, i'd not expect the fs to detect anything unless you use an fs with full data journaling (e.g. ext4 has such a mode, Btrfs doesn't). Do you have Btrfs logs of the previous corruption? Which kind of file/software did that happen to, SQLite or smth else?

[ell1e]

Oh it can happen to any large file that you copy on there really, it's probably just SD cards occasionally bit flipping and being SD cards. I wouldn't recommend using SD cards with anything but a filesystem with checksums. It leads to a dmesg message pointing out the checksum error and the open() fails with an EIO or some similar errno value. If you write like, 50GiB of files then the chance of one of them having a bit flip isn't that small on an SD card, I tried multiple card vendors too.

[iequidoo]

So this time it's definitely not a bit flipping. I'd suggest you try another fs, say, ext4 in the default data=ordered mode and see what happens (i'd expect the same corruption). Btw, how often does this reproduce? Then you can mount the fs in the data=journal mode and make more tests. https://serverfault.com/a/914466 explains how data journaling helps to maintain data consistency w/o write barriers working/enabled. But you should make sure the journal is big enough, 102400 blocks is the maximum size apparently.

At least some experiments may help to understand how to use Delta Chat on such devices or falsify assumtions about possible reasons of the problem.

[ell1e]

It only really reproduces every 2-3 weeks or so, the Delta Chat getting stuck event, so it's not very often sadly. I use this device a lot so I can't easily switch out the filesystem. I had ext4 at some point long in the past, but I didn't use Delta Chat yet around that time.

[iequidoo]

I guess, you switched to Btrfs because of ext4 lacking data checksumming? Then Btrfs is better but also not ideal it seems. What you can do is to somehow script Btrfs auto-snapshotting (with rotation of old snapshots) and in case of a hard shutdown just roll back to some sufficiently old snapshot so that all data is already synced by your SD card there. You'll lose some amount of recently fsync()ed data, but anyway your SD card apparently doesn't guarantee the opposite and at least you won't get such a silent data corruption (it's actually silent because well, SQLite detects it, but other software doesn't necessarily). Still i'm not sure that will guarantee fs metadata consistency because if Btrfs moves around some metadata at the moment of a power loss, it may get corrupted, but at least not silently and i guess Btrfs should avoid that (i'm not familiar with the Btrfs source code, but usually doing so is a bad idea).

[ell1e]

I guess, you switched to Btrfs because of ext4 lacking data checksumming

Correct! My guess would be btrfs likely avoids every kind of silent corruption, other than perhaps actually guaranteeing an fsync() goes through if the SD card doesn't reliably report it, which I doubt that it does.

[ell1e]

I have an interesting update. I have since seen this happen multiple times, but it seems like the actual database image being malformed was a fluke. Instead, most commonly the core seems to be stuck (and I mean for all practical purposes really stuck, I've had it sit for 40 minutes!! once and this continued on and on) with the log being filled with an endless stream of entries like these:

2025-07-01T18:42:20.964Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message fcada117-13f8-43bf-8431-ed95d77f6aee@localhost that we have seen before."
2025-07-01T18:42:20.972Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 5587968d-9a3f-4e2c-9b16-2b8df13cde05@localhost that we have seen before."
2025-07-01T18:42:20.977Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 0fcd52b4-725a-4b34-b558-19f6a0f055e3@localhost that we have seen before."
2025-07-01T18:42:20.984Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 10d11731-a6e4-4884-9b32-407aa1819837@localhost that we have seen before."
2025-07-01T18:42:20.995Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 8ec5ac8b-d9a9-44ba-8b92-0faa92a88f38@localhost that we have seen before."
2025-07-01T18:42:21.005Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 79db4b74-1d14-4f36-91d4-2d115953280f@localhost that we have seen before."
2025-07-01T18:42:21.014Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 5b150551-6507-4b3a-a9cb-d6401c537deb@localhost that we have seen before."
2025-07-01T18:42:21.026Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 36fdfb8b-b65f-4665-aa10-d828cc3536b2@localhost that we have seen before."
2025-07-01T18:42:21.035Z    core/event              INFO    ""  1   "src/imap.rs:627: Not moving the message 02b132cb-2e85-49bf-98da-431679e12d85@localhost that we have seen before."

Sometimes it wakes up again after a while, I've seen it at some point be in this state for only like 10 minutes, but sometimes it either doesn't or it's so late that for all practical purposes it makes more sense to nuke the install and restore from a backup.

[iequidoo]

Btw, #6877 removes this logic and Core will try to move all messages from the Inbox and Spam folders as if it sees messages for the first time. But i don't think that will resolve the problem. It seems that all messages in the folder are prefetched again from scratch. What folder is it?

[ell1e]

Okay, here's details about my setup:

1. I'm guessing what it's fetching from would be the "DeltaChat" folder, I always have the checkbox set to always use the "DeltaChat" folder only.

2. I don't use any deeper subfolders or any regular mail activity inside DeltaChat, only outside in the other folders (that aren't nested in "DeltaChat") that it's not supposed to be watching.

3. The IMAP account is shared with a classic mail account, but the DeltaChat traffic uses a separate alias and goes to the subfolder via a server-side rule that moves the messages into that sub folder too, if DeltaChat at any point doesn't do that already.

4. Any sort of auto deletion in DeltaChat is disabled, since the warning text so clearly tells me not to use it for just a subfolder. Instead, I clean up the folder using Thunderbird sometimes. Multiple DeltaChat clients share this setup and account, so messages are retained on the server until I delete tthe old ones after usually around 20-30 days.

[ell1e]

I wonder if there is still something useful I could investigate, or perhaps some idea to work around this?

The problem is less so that it redownloads message headers at some point, but that it seems to download them for a long time and compare and just take a really long time to do it for so many messages. So I feel like this is made more of an issue simply because it can take multiple hours. Afterward the client usually becomes awake again, therefore it's a sort of transient and in theory rather benign issue.