A very uninformative progress update: Mailpile 2?

#1

Hi everyone!

I posted a while back that I was planning to restart development this summer. I just wanted to inform y’all that I have done so, but it has proven to be a much more up-hill battle than I had hoped. I haven’t got much to show for my efforts yet.

My current main priority is to get the project back in a state where people could participate and contribute. The biggest blockers there, IMO, are the Python 3 issue and the fact that our “web framework” is crufty and confusing.

So I started working on Python 3 support, and realized that it’s quite hard - the syntax issues aren’t a big deal, but due to the way Python 2 and Python 3 differ in handling of strings vs. bytes, I need to review pretty much every line that manipulates data and decide which should be which (strings? bytes? what encoding?).

The effort involved is close enough to a rewrite, that I’m basically treating it as such, copying one source file over at a time, reading it and rewriting what needs to be rewritten. I’m not trying to resist the urge to clean things up in the process.

I’m doing this in “stealth mode” right now, so I guess this entire post is a tease about vaporware - I’m not showing my work yet, because I just want to focus on the tech stuff and I’m introvert enough that I find coordinating and explaining to be a bit of a burden. I’m also not sure what I am doing is even a good idea, so I’m going to give it a try for a couple of months and reserve the right to just throw it all out and return to the current Python 2 code.

So that’s where I’m at. We’ll see how it goes!

#2

Thank you for the effort! You will quickly find the typical patterns in your code, which makes this work easier.

But does’t it make more sense to merge/reject the PR first?

#3

I don’t know!

I haven’t figured out how much of my time I want to spend on the current Mailpile code, vs. the rewrite/port. Reading and formulating opinions about pull requests is work, deciding whether to then merge them into my new code is ALSO work.

I’ll be honest that I have had so much on my mind lately, that I have just been ignoring it. I know this sucks, but my personal life has only left me with very limited time for tech work and I just can’t do everything. I’m sorry.

#4

Oh, okay. Didn’t think that the rewrite would be that thorough. Then, try to focus on the rewrite, I would say. After that, we see which PR/issues still make sense.

#5

I’d like to volunteer some development time on Mailpile. I’ve been searching for a mail client that meets my needs for years and I think it’s just time that I try to help build one :slight_smile:

I was going to try to pick up some of the low hanging fruit issues from GitHub as a starting place. Is that a useful thing to do? Or should I wait for a v2 branch? Or is there something else that I could help out with?

Thanks for your work on this project, it looks awesome :slight_smile:

#6

Just checking in! Hello!

I have been doing a lot of work on this, and I have to say it is going very well.

I expect it will still be another month or two before I have something to show people, but I am laying some very sold foundations and porting code over as I go.

There are some fundamental philosophical/design changes I can tell you all about though, in the meantime!

Mailpile 1 preloaded a lot of data into RAM to accelerate search results. This worked as intended - my Mailpile (v1) now has over 1 million messages on file, but results are still fast. This had serious downsides though. In particular, the startup time for the app goes up with time, increasing linearly as more mail is processed. It now takes me around 10 minutes to restart my Mailpile, which is frankly unacceptable. The other downside is RAM usage; most of those million messages I really don’t care about and Mailpile uses gigabytes of RAM to allow fast access to searches involving them. This isn’t the worst tradeoff in the world, but it’s also not the best.

I am taking a different approach to the metadata index in Moggie. Moggie no longer loads the data directly into RAM on startup, instead it just loads some compact indexes and mmap()s the data files. Startup becomes almost instantaneous, and the plan is to rely on the operating system kernel to cache frequently used metadata in RAM, instead of doing so ourselves. The data structure itself is designed to facilitate this: recently received mail and old mail will (over time) occupy different files and different regions of disk, allowing the kernel to more easily cache the data we are most likely to care about and ignore the rest. We waste some disk space to facilitate fast in-place edits/updates, but make up for it by using a tighter encoding scheme than Mailpile did.

Overall this will make Moggie’s search performance a little more erratic (especially if people are using spinning-rust storage), but we should make up for it by not wasting CPU and RAM on data we don’t intend to use. The code itself is also smaller, Moggie just does less work in Python space, offloading more to the OS kernel. Which is also good for performance.

I’ve been having a lot of fun working on this.

I was testing the code last night, and combining the new metadata index with some carefully optimized mbox loading code I also wrote recently, Moggie can build a fresh new metadata index from scratch many times faster than Mailpile 1 could merely load its pre-calculated index from disk.

All of which is to say, progress is being made, and I am becoming more confident that my decision to start from scratch and use Mailpile 1 as a source of code snippets I can use or discard as I like, is feeling very good right now.

#7

Oh yeah, the code-name for Mailpile 2, is Moggie. Which is homage to the mutt mail client, which I see as inspiration for how light and fast I’d like the app to feel. A mutt is a mixed-breed dog, a moggie is a mixed breed cat. :smile:

#8

Great to hear. Found Mailpile online, got very excited, tried to install on Ubuntu 20.04/Debian 10 and :woozy_face:! But I am glad you have started work on v2.

Hope to meet Moggie soon.

Cheers!

#9

Another quick update: I got a bit bogged down on Mailpile/Moggie work last November. Things had just gotten a bit complicated and I lost my focus.

To get back on track, and to make sure I was able to actually deliver something useful within a modest time-frame, I decided to narrow my focus and work on releasing one of the Moggie-related innovations: the ability to “reset your password” without compromising (too much) security. This goal is to make it possible for a user to lose/forget their Moggie passphrase without irrevocably losing access to their encrypted data. There are of course tradeoffs here, security vs. reliability (one size does not fit all!), which I will have to explain at greater length in another post.

Anyway, that work is about ready to go public; I’m getting ready to push to PyPI and Github a proof-of-concept system called Passcrow, a generic system for community assisted secure password recovery. I’m pretty excited about it!

There is a new section on this forum for discussing Passcrow: https://community.mailpile.is/c/development/passcrow

#10

Not knowing about this, and refusing to have Python2 stuff on my servers, I started doing pretty much the same work on the weekends.

Mailpile is now not screaming errors at me on Python3, but its SimpleXMLRPCServer implementation is not accepting requests. I’m moving it to wsgiref, since it seems like the lowest effort solution to get it to work, and the most backwards-compatible API.

My failure is not seeing this discussion.

I don’t have any emotional attachment to my changes, so I don’t mind throwing it out in favour of concentrating efforts. I want to see Mogggie up and running as fast as possible. I have fairly decent Python knowledge and experience that can help. Please let me know where I can contribute my ~4h per week. I’ve seen many places that can be improved, mainly due to Python3 having a much friendlier library, due to the benefits that proper data structures can bring, with mypy’s validation, etc. But you know what is the most low hanging fruit with the biggest benefits.

tl;dr; give me some tickets to try and work on.

#11

Hi Alex!

Currently, the low hanging fruit for next-gen development is in Passcrow, which is how Moggie/Mailpile 2 will handle password/passphrase recovery. If you are interested, issues 3 and 4 are relatively straightforward: https://github.com/mailpile/python-passcrow/issues

I’ve assigned to myself the ones I already have in progress.

Edit: It probably makes sense to comment on the issues if you are interested in taking them on, I may then be able to provide further guidance, since I do have a bunch of opinions on how things should slot together. Also, THANK YOU. :smile:

#12

Hi Alex,

If I understand your last comments, you have ported Mailpile to Python 3 snd are not getting compile errors, but the Py3 SimpleXMLRPCServer is not working. Do I have that right?

Can you let us know how much you have had to modify source code and how much more needs to be done to make it work under Python 3?

Of course it’s important to keep focussed on the priorities defined by Bjarni, but sooner or later the jump to Python 3 has to be made, so if you have made any progress in that direction it’s important not to lose it.

#13

Thanks for the update, and for all of your work on this.
It is a very worthy and much needed product.

#14

Another very minor quick update: I’ve got good momentum on Moggie again.

I can now use the app (in terminal mode, I’m postponing porting over the web interface still) to read mail! This is a fun milestone because it means I’ve got basic mailbox and e-mail/MIME parsing working. This paves the way for porting over some of the PGP logic, or starting to feed data into the new search engine. Exciting!

I’m also using the urwid Python library to make a “GUI” in the terminal this time around, which is turning out quite well and lets me quickly prototype different workflows/UIs without getting distracted by browser compatibility or making things look pretty.

#15

I’m not religious, but I’m praying for a stable Moggie release sooner rather than later as the state of open source webmail is utterly abysmal and it makes me sad :frowning:

#16

Another update!

I’ve posted a very simple feature-roadmap for Moggie, to this Github issue: https://github.com/BjarniRunar/moggie/issues/1

It’s a somewhat daunting list, and it’s maybe a little deceptive in that it makes it look like almost no progress has been made, since most of my efforts have been building low-level foundations so far. I’m actually quite happy with how the project is progressing. It’s fun to work on!

I am still not asking people to test or provide feedback, at the moment that would just be distracting. But I want y’all to know that work is happening.

#17

Time for another update!

The rewrite of the search engine itself and low-level foundations is progressing really well. It’s slow going, but I am very happy with the results. I posted a thread on Twitter with some thoughts and performance numbers: https://twitter.com/HerraBRE/status/1561319364493713409

One of the things I didn’t mention on Twitter, is I’ve decided that I may as well make sure Moggie implements enough of the notmuch command-line interface, that it can be used as a drop in replacement for people who don’t want to convert their archives to Maildir, or who want access to some of the other features I will be developing. This kind of API compatibility is a pretty easy shortcut to having multiple frontends relatively soon; I’ll probably make a proof of concept fork of https://github.com/akissinger/dodo/ that can use Moggie as a backend.

I’m also in discussions with NLNet about getting some funding for the project, and that is sounding very promising (we are reviving an old grant application).

#18

Time for another update!

September was good for Moggie. I polished off our old NLNet grant application, updated it for Moggie and got to work. The results are chronicled here: https://github.com/BjarniRunar/moggie/issues/2

To make a long story short, Moggie is now usable as a (still crude) custom backend for the SearX meta-search engine (https://github.com/searx/searx), if people are self-hosting SearX and want to search their private e-mails using the same interface as they search the public web. I don’t actually know if this has practical utility for anyone, but it was a useful goal to work towards, since it made me push the search engine partitioning features out of “alpha” and into a usable state.

My next milestone will be to compete the notmuch compatibility work (see https://github.com/BjarniRunar/moggie/issues/3). This will mean finally porting over Mailpile’s old GnuPG support - or replacing it with something equivalent. So this task might escalate a bit!

I will also be looking into applying for more grants, to extend the project runway a bit.

#19

Been a while since I posted! Work is ongoing, as time allows. I got a bit distracted from completing the notmuch integration, but I’m making random progress in all directions as time allows - most recently I have done a lot of work on actually generating e-mails for sending.

I have also started posting on the Fediverse (Mastodon) about my work on this, so if people want a much more frequent brain-dump, follow me at https://floss.social/@HerraBRE (I think there are also RSS feeds and things, not quite sure.)

I’ve also applied for funding from NLNet for moggie. We shall see whether I get it! If I do, that will come with a bit of a roadmap and some concrete milestones which I will tell folks about on here. And maybe it will be time to retire this megathread and start posting new ones! :smile:

#20

Final update here! I got that grant, and there is a plan.

This is the thread to watch now: A grant from NLnet: A Mail Client in Six Steps