Which semester in ECE is toughest and why?

I think you will find that the 3rd and 4th sem are the toughest for any branch as this is the time you are first introduced to the core subjects of your own branch. That said, different students will find different subjects to be very difficult.

For some, it's advanced mathematics and for others, it is engineering drawing, right in the first year! LOL!

Have you had Digital Signal Processing yet? Some of my friends found it super difficult.