1. 'Engineering For Change In India' - Mega Contest, Grand Prizes!
    >>Click For Details<<

Help Us Catch Errors - Help CE Become More Powerful!

Discussion in 'The Café - General Discussions On Everything' started by Kaustubh Katdare, Feb 27, 2014.

  1. Kaustubh Katdare

    Administrator

    Engineering Discipline:
    Electrical
    We're fine tuning our server and setting up systems to deal with the traffic spikes. At this time, we've put CE on the new system and that might mean there could be a few errors thrown up at random.

    I request all of you to keep an eye on the site - and if you encounter any error; please copy the text and send it to 'admin [at] crazyengineers [dot] com' as soon as you can. It'll help us make sure that the system performs well and everyone benefits.

    We look forward to everyone's contribution and support!

    Thanks!
     
    • Like Like x 1
    #1
  2. SarathKumar Chandrasekaran

    Engineering Discipline:
    Automobile
    I dont feel or see any errors. CE has been running flawlessly and quick. If we found anything wrong, I am sure we will post the errors.
     
    #2
  3. Kaustubh Katdare

    Administrator

    Engineering Discipline:
    Electrical
    A note about yesterday's downtime -

    Last night (IST) about 4 articles in CrazyEngineers VoiCE were trending in various countries, resulting into heavy inflow of traffic. We went from about 800 users online to 6500 and that's where things went wrong.

    A similar problem that had happened last time led us to upgrading our database delivery hardware. But this time, PHP memory just couldn't handle requests from so many people visiting us simultaneously.

    The fix; however was not very easy. Finding down what part of site was eating up all the memory was difficult and involved lot of trials and errors. CE's become a complex structure (backend) and what you see on the front is several different system working in sync. It took us about 5 hours to figure out that 'Engineers Online' section displayed in the sidebar was the real culprit. While we employ lot of caching; that section isn't cached; and almost blew up php memory limits.

    The only fix we figured out was not to display that section on the articles and discussions pages; but show it on our front page, community page and individual forum section pages; until (maybe) we upgrade hardware. Another fix would be to manually disable that section whenever traffic shoots. We'll have to figure out a better solution.

    Thanks to all of you who reported errors on the site through SMS ( @Anil Jain, Our Authors ) and several others who sent emails - we were able to track down the problem.
     
    #3
  4. Anand Tamariya

    Engineering Discipline:
    IT
    You might want to try Amazon EC2 cloud server infrastructure to handle such peak loads.
     
    #4
  5. Kaustubh Katdare

    Administrator

    Engineering Discipline:
    Electrical
    We're on the cloud anyways. I think no cloud provider allows for real time scaling of resources. Our current servers are capable enough and should have been able to handle traffic easily. Especially the web server and database servers showed no load at all.

    Setting up PHP memory limit is a double edged sword. I'm still trying to figure out what should be the most optimum setting.
     
    #5
  6. Anand Tamariya

    Engineering Discipline:
    IT
    I suggest you don't jump guns without discovering EC2 a bit http://aws.amazon.com/ec2/ .
     
    #6

Share This Page