Help Us Catch Errors - Help CE Become More Powerful!

Replies

• 

  Sarathkumar Chandrasekaran

  I dont feel or see any errors. CE has been running flawlessly and quick. If we found anything wrong, I am sure we will post the errors.
• 

  Kaustubh Katdare

  A note about yesterday's downtime -
  
  Last night (IST) about 4 articles in CrazyEngineers VoiCE were trending in various countries, resulting into heavy inflow of traffic. We went from about 800 users online to 6500 and that's where things went wrong.
  
  A similar problem that had happened last time led us to upgrading our database delivery hardware. But this time, PHP memory just couldn't handle requests from so many people visiting us simultaneously.
  
  The fix; however was not very easy. Finding down what part of site was eating up all the memory was difficult and involved lot of trials and errors. CE's become a complex structure (backend) and what you see on the front is several different system working in sync. It took us about 5 hours to figure out that 'Engineers Online' section displayed in the sidebar was the real culprit. While we employ lot of caching; that section isn't cached; and almost blew up php memory limits.
  
  The only fix we figured out was not to display that section on the articles and discussions pages; but show it on our front page, community page and individual forum section pages; until (maybe) we upgrade hardware. Another fix would be to manually disable that section whenever traffic shoots. We'll have to figure out a better solution.
  
  Thanks to all of you who reported errors on the site through SMS ( @#-Link-Snipped-#, Our Authors ) and several others who sent emails - we were able to track down the problem.
• 

  Anand Tamariya

  You might want to try Amazon EC2 cloud server infrastructure to handle such peak loads.
• 

  Kaustubh Katdare

  Anand Tamariya

  You might want to try Amazon EC2 cloud server infrastructure to handle such peak loads.

  We're on the cloud anyways. I think no cloud provider allows for real time scaling of resources. Our current servers are capable enough and should have been able to handle traffic easily. Especially the web server and database servers showed no load at all.
  
  Setting up PHP memory limit is a double edged sword. I'm still trying to figure out what should be the most optimum setting.
• 

  Anand Tamariya

  Kaustubh Katdare

  We're on the cloud anyways. I think no cloud provider allows for real time scaling of resources. Our current servers are capable enough and should have been able to handle traffic easily. Especially the web server and database servers showed no load at all.
  
  Setting up PHP memory limit is a double edged sword. I'm still trying to figure out what should be the most optimum setting.

  I suggest you don't jump guns without discovering EC2 a bit Secure and resizable cloud compute – Amazon EC2 – Amazon Web Services .