Failed to open database (I guess)

#1

Mailpile (apt package) is running on a Debian server. Yesterday, I stopped the server, restored an old snapshot, and finally changed my mind so I went back to the volume that was being used before stopping the server. Therefore, I’m mentioning the snapshot just out of completeness, but it is not in use and my server should be running just like it was before I stopped it.

However, Mailpile now fails to load my data. It is correclty served on my subdomain (I configured it through nginx), everything like usual in that regard, and I can log in and see my three e-mail accounts, but there is absolutely no content except from two conversations:

image
image.png1892×894 62.7 KB

There is not even any sidebar, probably because something fails with loading the database associated to my e-mail accounts I guess?

The accounts are there:

image
zVVj.png841×292 35.5 KB

In the Network page: no activity. In the Events logs: Mailpile seems to be doing stuff.

Finally, this is what it looks like when running mailpile on the server (I usually run it with a service and have it served to a sub-domain, but I tried the normal way here):

Nothing abnormal I think.

If I type login, I get this but it seems normal too I beleie:

zVV9.png1138×753 229 KB

Any ideas? Could it be a permissions issue, and if so how would it have changed? /opt/mailpile is owned by mailpile:mailpile.

#2

I restarted Mailpile once and now even those two unreadable conversations in the screenshot above are gone. There is just nothing except my email accounts. Note that I cannot edit their settings: the pop-up window that should show the settings doesn’t fully opens, I only see it appear and show its title, but there is no content below.

#3

Just a guess: the snapshot function is not doing exactly what you think it is doing!

The following is a shot in the dark, but for what it worth:

The mail index is stored in mailpile.idx. The basic data on mailboxes and tags is stored in mailpile.cfg. Mailpile keeps multiple backups of both these files e.g. mailpile.idx.5 and mailpile.cfg.5. You could rename and copy files (with Mailpile not running!) to restore one or both from one of the backups, then run Mailpile. I’d try mailpile.cfg first. If you get the mailboxes back, but they are empty, try waiting a while for a rescan.

Some mailbox config data is also stored in files pickled-mailbox.nnnn which are not as thoroughly backed up.

#4

Thanks for your answer @JackDca. I’ll try your suggestion, that seems relevant! Perhaps the data was corrupted when turning off the server if I forgot to stop Mailpile first. However, what seems weird is my .cfg and .idx files have modification dates back in March and April 2019.

About the snapshot: you are probably right, but in the end I did not use it. I reverted to the volume I was using before, and when I say that I came back to it, it was the actual volume as it was before I tried the other snapshot, not a snapshot of it. See it as a different drive with everything running on it: I just unplugged it, tried the snapshot on another drive, and decided to replug the previous drive. But weirdly enough, Mailpile does not work anymore now.

#5

I restored both mailpile.cfg.1 and mailpile.idx.1 as mailpile.cfg and mailpile.idx, respectively, but I still have the same issue unfortunately.

#6

Hello @Kabouik

That is distressing. The ancient modification dates are particularly strange. I have myself worked on the code that saves mailpile.idx periodically (PR #2063). That was more than a year ago so I don’t remember the specifics, but considerable effort is made to avoid data loss even in the case of a “dirty” shutdown (in fact I’ve done that quite a few times myself on my local computer without catastrophic results).

Can I ask a few questions -

  • When did you first create this Mailpile instance?
  • Are the dates on mailpile.cfg.1, mailpile.idx.1 etc also back in March and April? If so how much time span between the oldest and the newest/
  • Is it possible that Mailpile was running without interruption from April 2019 to the time you most recently turned off the server?
  • How big are the mailpile.idx files? Mine is about 30 MB for about 60,000 emails - about 500 bytes per email. How does the file size compare to the number of emails that you had stored?
  • What does the mail subdirectory look like? It should have a subdirectory for each mailbox, each in turn containing subdirectories cur, new, tmp, which in turn contain a file for every email in the index (more of less). Are the dates on any those files current?
  • I’m not sure how the VPS works - never used one myself. Is there is limit on disk space? If so, what is it?
  • Is there any chance that the mailpile homedir (data directory) was moved at any time? That is, did the absolute path to the directory change at any time? There are some problems with that - see Issue #2233.

I’d like to make some effort to understand what has happened here. I’ll be out of touch for most of the next four days. but if you could answer these questions, I would appreciate your input.

#7

If mailpile.idx has a modification date in the distant past, then I think it’s safe to assume that you’re not looking in the right directory.

Mailpile will (unless instructed otherwise using environment variables) store data in the home directory of the user running the Mailpile process. Nothing in our default setup uses /opt/mailpile, so if you have content there, then you’ve done something unusual.

#8

That is actually something I had in mind after seeing @JackDca’s answer and the modification dates in /opt/mailpile, the first question I was going to ask was about the working directory being used by Mailpile when installed from the Debian package. I was using Mailpile from git before, and /opt/mailpile was the directory in use. When trying to restore old .idx.1 and .cfg.1 files, I just assumed I should look into the same directory for the Debian package, but already started to doubt that when I saw it had no effect.

I haven’t had the time to look into it since the last time, which is also why I didn’t answer to Jack yet.

I know running Mailpile from a service file is not officially supported, but below is the content of the mailpile.service file I use to run Mailpile, which shows it is indeed run as my main user. Note that I left comments with the lines I was using when I was using it from git:

[Unit]
Description=Mailpile Server
After=syslog.target
After=network.target

[Service]
Type=simple
User=kabouik
# Group=kabouik
WorkingDirectory=/home/kabouik
ExecStart=/usr/bin/mailpile --www= --wait
## Above: settings to run the mailpile Debian package as a service.
## Below: settings for running mailpile from the git folder in /opt/mailpile.
# Type=simple
# User=mailpile
# Group=mailpile
# WorkingDirectory=/opt/mailpile
# ExecStart=/opt/mailpile/mp-virtualenv/bin/python mp --www= --wait

# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=300

[Install]
WantedBy=multi-user.target

I’ll have to check into /home/kabouik/.local when I get a minute. Thanks.

#9

Hooray. mailpile.cfg.1 did not work either, but mailpile.cfg.5 does. Now I just have to find exactly what is the most recent working backup.

Thank you again to both of you.

[Edit] mailpile.cfg.5 it is then. All the other backups fail with the same issue, weird. Perhaps Mailpile kept making new backups all that time (I was stupid enough to let Mailpile run), despite failing to load a correct .cfg file, in which case it replaced my recent backups, and I was lucky enough that the fifth and last one was left untouched!

#10

A lot of different actions will trigger a change to the config or to the index. That’s why we create multiple backups… glad things are looking better for you!

#11

Good to hear there’s progress @Kabouik.

Just to add to @BjarniRunar 's comment - the default data directory is /home/[user]/.local/share/Mailpile/default.

When running from the command line you can use some other subdirectory of .../Mailpile by setting an environment variable before running Mailpile e.g.
export MAILPILE_PROFILE=notdefault
will make Mailpile use
/home/[user]/.local/share/Mailpile/notdefault.

(Handy if you need to have multiple instances of Mailpile for testing purposes.)

Might be helpful to see if there is anything in /home/[user]/.local/share/Mailpile.

Good on @BjarniRunar for noticing that you were finding the files in /opt - I missed that!

#12

Indeed, I edited my previous post when I realized I forgot about hidden folders in /home/[user]. Please disregard the email notification you might have received when I replied before the edition. :>

Good news usually come with bad news though. My emails do not go past the Outbox anymore, yet I am using the right credentials (I re-entered my password just to be sure) and smtp settings (they work in Mailspring). Mailpile manages to index new incoming emails, the issue is just on the outgoing side.

Here is what I get with one of my accounts (notification saying “Failed to connect to smtp.domain.tld”):

{
 "data": {
  "auth": true, 
  "delivered": 0, 
  "host": "smtp.redacteddomain.fr", 
  "last_error": "Failed to connect to smtp.redacteddomain.fr", 
  "last_error_details": {
   "connection_error": true
  }, 
  "mid": "UD8", 
  "proto": "smtptls", 
  "recipients": 3, 
  "sid": "M/1xBm40"
 }, 
 "date": "Tue, 20 Aug 2019 15:05:55 -0000", 
 "event_id": "5d5c0bdb-001ff-7bb5", 
 "flags": "i", 
 "message": "Could not send mail to myself.myself@redacteddomain.fr", 
 "private_data": {
  "myself.myself@redacteddomain.fr>someone.else@redacteddomain.fr": false, 
  "myself.myself@redacteddomain.fr>someone.else2@redacteddomain.fr": false, 
  "myself.myself@redacteddomain.fr>myself.myself@redacteddomain.fr": false
 }, 
 "source": ".plugins.compose.Sendit", 
 "ts": 1566313555.845003
}

And if I try another account afterwards, I get this (this time, the email doesn’t even go in the Outbox, I just get a red notification saying “Failed: mail”):

{
 "data": {
  "elapsed": 13, 
  "output": "json", 
  "ui": "HttpUserInteraction"
 }, 
 "date": "Tue, 20 Aug 2019 15:03:37 -0000", 
 "event_id": "5d5c0bc9-001dc-7bb5", 
 "flags": "c", 
 "message": "mail: Failed: mail ", 
 "private_data": {
  "data": {
   "_method": "POST", 
   "attach-pgp-pubkey": [
    "no"
   ], 
   "bcc": [
    ""
   ], 
   "body": [
    "Test2019"
   ], 
   "cc": [
    ""
   ], 
   "csrf": "(SUPPRESSED)", 
   "encryption": [
    "none"
   ], 
   "from": [
    "Mathieu \x3cmyself@otherdomain.tld>"
   ], 
   "mid": [
    "UD9"
   ], 
   "subject": [
    "Test"
   ], 
   "to": [
    ""
   ]
  }
 }, 
 "source": ".plugins.compose.UpdateAndSendit", 
 "ts": 1566313417.348221
}