Please read this entire post as it contains very important information.
Right, where are we at with this SLOW database?
Our new super fast database server has arrived, and basic setup has been done.
This is also one of the main reasons for this post….
Due to the physical location of the current main Imnica Mail server, we will need to move it so that it is in the best possible place for the fastest possible speed to access the database server.
To do this, we need to physically turn off the main server and re-locate it to the correct position.
I am assured by the hosting company that this should take no more than 15 minutes to do.
There is a BUT though ….
They can ONLY do it today at around 2PM US Pacific time, which is GMT -8.
Use the following URL to covert 2PM PDT to your local time: http://www.timezoneconverter.com/cgi-bin/tzc.tzc
Ok, so, what after the move?
After the move we will be doing some HEAVY testing with our development server talking to the main new database server. This testing will take place Friday and Saturday, and will have ZERO impact on the production systems.
We will fine tune the process of the switchover. Then, on Sunday, if we’re happy with all the testing, we will stop the web server on Imnica Mail, take a backup of the database, copy the backup to the new server, restore the backup, run some validation tests, change the config on the main site to talk to the new database server, and then turn the web server back on.
All of that should be less than 30 minutes. Exact times will all depend on the full database size at backup. We’ll get a pretty close idea during our testing tomorrow and Saturday.
If all has gone according to plan, and everything is done, we’ll have a MUCH faster site, and email will also send MUCH faster than they have now!
What about Support?!
Support right now is … Let’s just be polite, and say … LACKING. Yeah, I know, many of you will have MUCH stronger words to describe current support levels.
We are fixing this, and I promise you, with the new support staff, one of which is our new VP of Customer Care, you will see support levels like you have only dreamt of!
We are WAY behind, with a TON of tickets waiting to be answered. They will ALL be answered, and SOON.
Once the backlog has been caught up, barring some other catastrophe, we will never get more than 24 hours behind EVER again!
Support desk while site is down?
Being a young company, we have made some silly design decisions when we built everything.
One of those, was to put the Help Desk on the main web server. When this server however goes down, we’re stuck not being able to provide support.
We use the amazing Kayako Help Desk software, running on our own servers.
They have recently begun offering hosted services, where our whole help desk runs on their massive server infrastructure.
Once all our database issues has been sorted, we will be migrating to their hosted infrastructure, which will also mean zero down time on the Help Desk. Obviously we want this for the whole of Imnica Mail, and we’re working hard to get our UP time to 100%!
Even so, rather safe than sorry I say!
We have hit a road bump in our growth. Now, there are 2 ways to look at this …. Good, and/or, bad.
I prefer to see it as BOTH good and bad.
It’s bad from the perspective of: Our customers had to suffer sub par service for a while. This was everything from slow site, to slow email sending, to SLOW support.
It’s good from the perspective of: We have been growing SUPER fast, which our current infrastructure was not ready for.
So, we’re now upgrading our infrastructure to handle the new levels of growth, and explode Imnica Mail to much greater heights.
We also learned that our support organization is/was nowhere near ready to handle the level of growth, coupled with the issues we had. We’re adding more support staff, and more automated systems.
Basically, we learned a lot, which helps us to massively improve our whole setup, infrastructure, and company.
I will not lie to you and tell you that we did not lose customers because of the issues. We did. A lot.
We had many more, not only stick around, but, always have encouraging words to help us along, while we just got on and fixed the issues.
To the customers who left. We’re truly sorry, not only to lose you, but, to have failed you.
To the customers who stuck around, and even the new ones who may not be aware of some of the issues we had over past few weeks. Thank you so much for your trust. You will not be sorry!
Signing off now to start minor preparations for the physical move. If all goes well, by Monday, you will see an Imnica Mail faster than ever before! 😉
Richelo Killian, and my amazing team.Share