One of our servers was having repeated issues this morning. In each case, it was causing a denial of service for about 1/8th of our users for upto 10 minutes at a time. We are investigating the root cause. We apologize for the disruption in service.
We always have an engineer on support every day, weekday or weekend. Today is actually my support day. Timezones make things difficult though. This happened at about 5AM our time (Seattle). I was online till 1AM and then went to sleep. We also do have Aleksi in Finland to help with support but he is not online 24X7. So there are some times when it can go a few hours before we see something.
Actual outages lead to the system auto-recovering (as it did in this case), but it usually takes a few minutes each time for that to happen.
This particular case is because some one user’s app seems to be causing some kind of runaway computation on our server.
That should never happen, but clearly, there is a loophole somewhere that let’s it occur. We have to trace back to find out what app caused it and why. In fact, it could well be one of you guys on this thread :]. We noticed that each time the server was restarted the problem re-occurred.
LinkedIn
Twitter
