Castles OP in Castle Age?

I always struggle if an enemy builds a castle in my base in the early/mid castle age. The only answer are rams but the enemy already has units to destroy them. Enemy Pikes stand in the range of the castle to protect it from the rams. It is frustrating. Going imp is also very difficult in mid castle age if your eco get raided. Yeah, in Imp I could build Trebs/Cannons. But there is nothing to destroy castles in castle age from a safe range.