After few week of Mdaemon slow queue complain, 1 week of changing hardware test and 1 week of process tracing, mail queue is now stabilized.

Summarize these week study; there are few setting should help Mdaemon run faster

1. Offload Remote Queue to other server by setting delivery options [Must]
2. Use server 2008R2 with enough memory to cache more Metafile
   and calculation of enough memory = 4GB + (files count * 1KB)
   for example, total mail store with 4 million files should have 8GB or more
3. Registry tweak  (based on Performance Tuning Guidelines for Windows Server 2008 R2)
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
   NtfsDisable8dot3NameCreation=1
   NtfsDisableLastAccessUpdate=1
4. Disk defragment (no need often defragment)
5. Application disk volume alignment, Mdaemon Application ( cfilter dir), Local Queue and User folder should be on same volume
6. CPU – higher clock / better clock efficient instead of multi core
   a. we tested on 8 core 16 thread machine, it use only few thread
   b. even average load is below 50%, you will see 2.8GHz CPU perform better than 2.4GHz CPUI think Single CPU SandyBridge 2600K server will run faster than server with Dual Socket Xeon E5625

Offload Remote Queue to other server by setting delivery options

After offload Remote Queue to another Mdaemon server, you will see Incoming Queue run faster (less lockup)

Use server 2008R2 with enough memory to cache more Metafile

When huge amount of files and mail box are accessed frequently, server 2008 will cache Metafile to memory.

for example, Mdaemon folder have 4,385,755 Files, the usage of  Metafile can reach 4.7GB.

Also, more memory also can benefit windows read cache. Equallogic SAN HQ report show the read/write ratio change from 66/33 to 15/85 after system upgraded to 16GB memory. That mean 91% of read IO saved by system cache.

Application disk volume alignment

After tracing server with Sysinternals’ Process Monitor ( 3 minutes of event, data size 5.9GB!! ),

I found,

1. Mdaemon access disk in 4KB per Read/Write, it would be fine for mail sized in KB sale. However, some user send mail in Megabyte scale.
2. Mdaemon create Local Queues and Remote Queues by coping message from Inboundfor example, a email addressed to 5 local mailbox and 5 remote mail recipient with 3 different domain will do following
   5 times of ( make unique per recipient mail header + copy remaining message body in 4kB per block from Inbound to Local till EOF)
   3 times of ( copy message in 4KB per block to Remote Queue  + create *.rte to record target recipient)
3. when process Local Queue and found user have mail forwarding setting, it will create related Local Queue and Remote Queue.
   And same as Inbound Queue, working in 4KB per block pattern.
4. Local Queue will do Antivirus and SpamAssassin check, file will move to Cfilter\Temp during CFilter process
5. The final local mail will move to user mailbox

You will found that, file will  move between CFilter, Local Queue, and User Folder.
When the folder not in same disk, Mdaemon still calling filesystem MoveFileA and move between volume handled by system.
On server 2008R2, read buffer is 512KB and write block size is 64KB.

CPU – higher clock / better clock efficient instead of multi core

You will found that, upgrade from single Quad Core CPU to dual Quad Core CPU only have slightly improvement.

It is because, queue processing seems single thread (one thread scan folder, one thread move file)
However,  CFEngine, Spamassassin and  Antivirus seems running in multi-thread, so, messages waiting inside Local Queue mostly are MDAV & SpamD processed.

原本上星期已有構思，想寫一個插件加到MDaemon去將收到郵件的數量返影給MRTG/CACTI 等rrdgraph。
因為只有rrdgraph才可以有助分析Server 繁忙時間分佈。

找了一會, 想用MDaemon 的guicounts.dat 內數據用WebService Publish 出去.
再找了一會, 找到一個2006件的project MD-MRTG

雖然舊少少, 部份設定不是很好。但經一番修改後總算可以使用。
修改內容如下

1. index.html font size and graph size double (new target screen width > 1280)
2. Web Session folder path (non default location)
3. Graph Max Value of (POP3 session, Mail In/Out, Spam, Network Bandwidth)
4. Inbound count method support Hash directory (sub directory)

唔睇都唔知, POP3 Session / hour 可以上到28K、SMTP IN 也有近8.5K。
晚上原來有近半的EMAIL是垃圾。

繼上年YAHOO.COM.CN個Outdated Bogon BGP 事件, 同YAHOO的 technical support 交手以後. 今年又要面對 YAHOO.COM 的postmaster.

因為見到有幾封十幾MB的郵件在mail server loop 了很久，才發現 send 到 yahoo.com 經常發生drop connection 的問題.

大郵件發送到YAHOO.COM會無原因地被對方斷開。由於沒有 SMTP Status code*1，E Mail server 會即時重試另一MX*2。 YAHOO.COM 的DNS記錄總共有 8個 MX RECORD。如事者，一封20MB 的電郵每次會傳到YAHOO.COM 的 8個 MX IP，每個IP傳20MB資料。

我地Server 每半小時重試一次，24小時不能成功遞送才會放棄。

即是寄一封20MB的信件到YAHOO.COM會花了20MB x 8 x 2 x 24 = 7.68GB的bandwidth。平均全日0.7Mbps (7680MB * 8bit / 86400sec).

以Singtel 去海外每Mbps每月約$1,000計算, 一封信花了$1,000 * 0.7 / 30 = $23.

*1 Enhanced Mail System Status Codes - rfc3463
*2 MX = Mail Exchange - rfc1035

市佔率很大的spamassassin 其中一條rule在2010 出現問題.
由於這條是Default rule, 很多Server都受影響.

名稱是FH_DATE_PAST_20XX, 看來應該是2006加入的. 原意是日期大於2010年的就加3.2 到3.4分. 以防止有人用將來的日期Broadcast Spam.

理論上sa update會自動下載修補版本. 如果無就要自己手改了.

##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX Date =~ /20[1-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX The date is grossly in the future.
##} FH_DATE_PAST_20XX

改為

##{ FH_DATE_PAST_20XX
header   FH_DATE_PAST_20XX Date =~ /20[2-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX The date is grossly in the future.
##} FH_DATE_PAST_20XX

又或Disable成條RULE

score FH_DATE_PAST_20XX 0.0

Spamassassin bug list
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6269

SpamAssassin Rule: FH_DATE_PAST_20XX
http://wiki.apache.org/spamassassin/Rules/FH_DATE_PAST_20XX