Both email and web load remains stable and healthy. Customers should not be experiencing latency in relation to this System Incident.
(mt) Engineers continue to analyze the data gathered during their investigation in order to determine the root cause of the symptoms experienced while this incident was active. When a conclusion has been reached regarding this incident’s root cause, an update will be published.
The efforts made by (mt)Engineers have fortunately reduced email queues and service levels back within normal operating limits. Delays with email delivery should no longer be present. We are taking further steps to address any blocklist issues while we continue our work to investigate the root cause of this incident.
Engineers do not anticipate email latency to return. We will provide information as developments are made. Please check back for an update by 2:00PM for before.
As of about 9pm last evening, (mt) Engineers successfully completed the password change operation that we previously outlined. If you would like additional details on what took place and why, please visit this special Q&A article from our KnowledgeBase:
http://kb.mediatemple.net/questions/1807
In the course of changing the appropriate passwords, we also took the liberty of opening a brand new Support Request within each affected account. This was done to make troubleshooting any related issues easier for all involved parties. If you have additional questions or need assistance with any issues that have arisen due to the password update, please respond to that specific Support Request or call us at our toll-free number, 877-578-4000.
Moving forward, (mt) Engineers will continue the process of investigating the root cause of this Incident. As always, we will update this page as more information becomes available, or if we determine that any other actions are necessary.
(mt) Engineers have now determined that email blocklisting seen by (gs) Grid-Service Cluster.02 customers is not related to this incident. If you are experiencing a blocklisting issue, please submit a new Support Request and it will be addressed individually.
The next update regarding this incident will be made as information is available or no later than 2PM today.
We have had reports of email related problems on Cluster.02. Some customers are experiencing email delivery latency, and others have reported that emails have bounced due to blocklisting. We believe these problems may be related to the core problem that this incident stems from. (mt) Engineers are taking action to relieve these problems, as well as investigate the correlation between these new issues and those related to web services seen earlier.
We will provide a status update regarding these matters within the hour.
At this time (mt) Engineers have upgraded the vast majority of storage segments on (gs) Grid-Service Cluster.02 with additional resources. While website load-times have stayed healthy through the night, they seem to show varying degrees of load to web services currently. Our engineers will continue to closely monitor the service levels throughout the day to observe the effects of these upgraded servers during “peak” times.
Barring any unforeseen complications, we will only provide an update to this incident once our investigation of the root cause is complete.
As of 6:51PM PST, our system engineers have restored database availability for all customers located on Cluster 02. We were able to confirm that the database host machine was online throughout the incident however the machine was not available to our public networks. The network issue was isolated after this was diagnosed and all services are currently online. Please note that no MySQL data was lost or corrupted as a result of this incident; the issue was limited strictly to our network.
Thank you for your patience while we worked to resolve this matter.
As of 4:58/PM PST, (mt) Engineers are currently in the process of investigating an ongoing issue, impacting some customers residing on Cluster 02.
Symptoms may include:
* Inability to access database driven websites.
* Inability to access phpMyAdmin.
As further progress is made and information is gathered, we will continue providing regular updates on this status page.
While our investigation into the root cause of this issue remains ongoing, our Storage Team is starting to conduct emergency capacity planning and implementation measures to help mitigate the problem. A large portion of this entails adding additional resources to the service. This should help bring the service levels of the affected Storage Segment back within better limits.
We anticipate that these additional resources will be mounted and in operation by the end of the working-day today. We will provide a status update with our progress by 6PM this evening.
Our Storage Team is currently performing ongoing diagnostics and working to resolve this System Incident. Unfortunately, the cause of a problem is not always readily apparent and can take some time to identify and eradicate.
At this time we know the following:
We are working to bring things back to normal as quickly as possible. We thank you for your patience and will provide our next update as soon as additional information becomes available.
(mt) Web log including various communications.
Adapted from Tumblr theme tumblrist
by topherchris.
powered by Feedvolley.
