Let me give you a quick run down on what is happening at Imnica Mail right now.
About a month ago, sending of email started slowing down considerably.
We have spent almost every waking moment, working on this issue. We had 3 teams working on this.
One was operating system, one was database, and one was the main Imnica Mail application.
All three groups worked like crazy and came up with all kinds of ideas as to what MAY be the issue.
Nothing panned out.
A couple of days ago, someone from the operating system team said the disk access was off the charts, and the disk was SO busy that nothing could be written to it, which is why everything was so slow.
After some more digging as to why the disk was so busy, we found out that our newly installed super MTA, was the main culprit. Now, the software itself did nothing wrong!
Our application writes emails to a pickup directory. The MTA then grabs the mail from there, and writes it to it’s own spool directory for processing. Nothing wrong there.
The problem came from the fact that we were pushing WAY to much mail through, and the actual hard drives could not keep up.
We stopped the MTA, and everything QUICKLY improved.
Lots of down time.
Next, Murphy struck…
One of our hard drives began failing, which again slowed down the system. This was not a major issue, as all our drives are in a raid fault tolerant setup.
We replaced the drive. This was just 15 minutes of down time. When the system came back up, everything was running, but, as the new drive was being rebuilt, things were a little slower than it should be, but, not a lot.
A few hours after the drive finished rebuilding, the server cam to a screeching halt again!
This time, the actual raid controller failed!
The controller was replaced, and when the system came up, everything was running fine, and super fast.
Well, we thought all the issues were resolved, and that we just had to move the MTA to it’s own dedicated machine, and we’ll be fine.
We had several hours of bliss. Around 12 or so, before …
The server stopped responding again!
By now, we are very experienced at finding where things are slow, and it became very obvious that the actual database itself was now killing the disk as well!
We’ve now made some changes to the database to force it to run slower and thus not killing the disks.
Everything has been running smoothly now for several hours, including some BIG campaigns which went out.
The end of Imnica Mail?
By now, many people are thinking/wondering/proclaiming the end of Imnica Mail.
NOTHING could be further from the truth!
We are growing SUPER fast, and these issues just PROVES it!
So, are we just going to keep the current setup, with all database activity forced to run slower than it can/should?
We have ordered TWO more MONSTER servers!
Both of these will have three times the amount of memory that the current one has, and both come with the fastest drives available on the market today, which is twice as fast as the current ones.
One machine will be dedicated to the database, and the other machine will be dedicated to the MTA. This will leave the current main machine free to run ONLY the Imnica Mail application.
So, what is the timeframe on these new machines?
We’re not 100% sure on that. These are very specialized servers, and take a little while longer to source than the normal run of the mill ones.
With testing etc to make sure everything works 100% as intended, I would say around 2 weeks till we go live on them.
Some people will now obviously be concerned about their existing sending, etc, till the new servers come online.
Well, things are not as bad as they may seem. Prior to this forced slow down, a 100K campaign would run around an hour. Now, it runs between and hour and a half to two hours. For smaller lists, you may not even notice a difference!
We are keeping a VERY close eye on everything an will fix any issue that comes up QUICKLY.
Even though we have been having these issues, my developers were still working on Imnica Mail, fixing bugs, and adding features.
There are already a few new features, as well as a new WordPress plugin. There is also another new version due in a week or two, which will add more new features.
I will wait on all the announcements till after the next release, and will then send out a complete update email with all the new features, etc, as well as information on the new WordPress plugin.
List imports has not suffered in any way, as I have a dedicated group now dealing with imports.
Normal support has suffered badly though!
My sincere apologies for this. I will be working with my other support staff as hard as I can over the weekend, and into next week to clear the ticket queue as quickly as possible.
Richelo Killian and the Imnica Mail team.Share