mail archives slow

Gerard Beekmans gerard at linuxfromscratch.org
Mon Dec 4 12:47:33 PST 2000


> That's not what I want. Usually I just want the individual messages. I
> read them and delete them immediately after reading, but sometimes, when I
> want to look something up, I'd like an up-to-date archive. It's also very
> helpful if your mailfeed has problems for a while. The HTML version is no
> good for that. It's not uptodate and even if it was, it's a pain if you
> just want to read messages in sequence. You have to do it online and you
> have to click around for navigating all the time. A simple text file I
> could download for offline reading would be best in those cases. The HTML
> version is only nice for long-term archival.

The second best thing I can offer you is you using the downloadable mail 
archives. They are bzip2'ed mbox archives which contain every single email 
from the day the list started. I don't want to give smaller text files. the 
question then comes up: how much should that file contain. A week? A month? 
More? Less? So that's why you can download the entire list archive (for all 
lists). lfsdiscuss.bz2 is a big file (3.6MB at the moment) but well worth it.

No, i can't offer you real-time archives. That is just not possible. That 
means you want the server to rebuild the archive every time an email arives. 
It takes about 7 minutes at least to rebuild the lfsdiscuss HTML archive. So 
real-time is impossible at the moment. The mbox files can't be updated in 
real-time either. I don't want to give the webserver read access to the 
/var/mail/lfsdiscuss or /var/mail/<enter list name> files. So those files 
need to be copied to the FTP archive and bzip2'ed periodically. bzip2 takes a 
few minutes (the mbox file for lfsdiscuss is about 25MB so it takes a little 
while).

A last possible option is the digest. Every time an email arives to the list, 
the digest for that day is updated (and sent out at midnight with the sum of 
the previous day's messages for the list). I could perhaps give apache 
read-access to those digest files. I don't think this is a major security 
risc since there is no secret stuff in listar's digest folder. That way you 
can get today's digest when you want it and it'll be up-to-day (well it only 
contains the current's day's messages). And I could have listar save digests 
so they can be archived.

If I can have postifx store the messages for the lfsdiscuss user's mbox in a 
seperate folder (using procmail I could transfer the mails to lfsdiscuss' 
homedir which I can give read-access without security risc) and give apache 
read-access to that directory and file. That way you can get the big mbox 
file and it'll be always up-to-date. Once a day you can get a bzip2'ed 
version from the FTP archive.

As you see there are quit a few options. Give me some time please. I do need 
time to implement this but I'll try my best to accomodate your and other 
people's needs.

-- 
Gerard Beekmans
www.linuxfromscratch.org

-*- If Linux doesn't have the solution, you have the wrong problem -*-

-- 
Unsubscribe: send email to lfs-discuss-request at linuxfromscratch.org
and put unsubscribe in the subject header of the message




More information about the lfs-dev mailing list