We are experiencing SLOW sending, and this time it is 100% our own fault.
Please allow me to explain what the problem is, and what we’re doing about it…..
Some of the explanation below get’s a little technical. Please bear with us, as it’s all important.
ALL of the issues below contribute to the slow sending in general. I will however start with the most severe one.
What I will do, is list the issue, and then what is being done to address the issue …
1. Current mail is delayed by several hours, even though the system says it has been sent.
Explanation on how the system works 1st …
How the system works is that our main application writes all outgoing mail to a special directory, which is then picked up by the mail transport agent, which is the main software doing the final delivery.
Once the mail is written to this directory, our system shows the mail as sent. The reason for this is as the pickup and sending by the MTA is usually VERY quick.
The problem …
As we have grown at an unprecedented rate, our sending has increased at a much faster rate than any of our planning has predicted.
That in itself is not the problem.
The problem is that our hard drives, although the fastest on the market, just cannot keep up with the speed of reading and writing, and the MTA starts falling behind when there is the smallest of glitches.
This is 100% squarely on our own shoulders for not planning and expanding accordingly.
What we are doing to fix this issue …
Other than pausing certain things, and nursing the system along to catchup the current back log, we have ordered a ton of new hardware, including some SUPER fast Solid State Drives (SSD), which is effectively just plain memory, and does not have any moving parts, so, will be faster than a speeding bullet when it comes to reading and writing.
This is by NO means our final upgrade, especially at the speed we are growing, but, it will last quite a while.
Time frame …
The hardware has already been ordered. Once it arrives, it needs to all be prepared, and then we need to take the main system down, install the new hardware, do some setup, some checks, and then bring the main system back into production.
Total time frame for all this to be done … Around 2 weeks max.
Down time will be 1 to 2 hours, and will be done at the least possible busy time of the week, which will most probably be on a Sunday night.
2. Somewhat slow sending outside of the current issue.
Even when we don’t have the current slow down due to hard drives not being able to keep up, we have still been slow from time to time, with mails being delayed for an hour or 2, and sometimes longer.
This slowdown usually happened during peak sending times.
This issue has 2 causes, and both are being addressed ….
2.a. The MTA was limited …
The issue …
We use a very powerful commercial mail transport agent (MTA) to help use deliver your emails.
We HAD the standard version, which was limited to 200 concurrent outgoing connections.
That sounds pretty low, but, in reality, we were able to deliver around 1 million messages per hour with that limitation.
The fix …
The fix is quite simple … Upgrade to the Enterprise version, which has NO limitations on concurrent outgoing connections.
Time frame …
This is done already! We upgraded to the Enterprise license last week Friday. You don’t feel the fix right now, as we’re struggling with the disks.
2.b. We don’t have enough sending IP’s.
The issue …
All AutoResponder companies like ours uses a whole bunch of IP addresses to send from. This is not to trick the receiving ISP’s at all. It is done as each ISP has a limit as to how many emails can be sent from a single IP at a time.
With our standard license of the MTA, as mentioned in the previous point, our IP’s were sufficient.
With the new license, and unlimited concurrent outgoing connections, we are still limited to how many mails we can send at a time from each IP address.
This is causing mail to go slower than it COULD.
The fix …
Get more IP addresses! Easy!
Time frame …
We already have the extra IP’s. We actually DOUBLED our existing number!
This is great news, but, there is a BUT …
We can’t just plug in in the new IP’s and start sending from them at full speed. This would just get them blacklisted and banned, and QUICKLY.
So, we need to follow a process, which is called warming. Basically, we start by sending a very low number of emails from each new IP address per day, and then we gradually increase that number every couple of days, over a period of 6 to 9 weeks.
Now, that time SOUNDS long, but, remember that we have double the number of IP’s, and even with a small number of emails per IP per day to begin with, this will still greatly increase our sending capacity to begin with!
Even though we have these new IP’s, we have not yet plugged them into the MTA, as we need to get the backlog caught up 1st.
Once caught up, hopefully within the next day or 2, we will then most probably load the new IP’s over the weekend and start warming.
We messed up BIG time by not being prepared for the growth we have been experiencing. NO EXCUSES AT ALL.
We are working around the clock to fix the situation!
We are truly and genuinely sorry for the current slow sending situation.
Richelo Killian and the Imnica Mail team.