How to press your advantage after feudal aggression in castle/imp?

In my journey to improve my ranked gameplay, I've noticed that I have a glaring weakness - I would harass the enemy with a feudal, seemingly have an initial advantage, make a forward base, and then have to delay my assault because they have walls. Unfortunately in several games my momentum ultimately ended up fizzling out as the enemy would just manage to get to imperial when all they've had is walls I just could not breach through, at which point they started counterpushing me and I would lose momentum at the game. I do believe I'm doing something wrong, but I'm not sure at all which is the correct response. How would one press your advantage in such a situation if your enemy is fully walled, possibly with castles/TC covering every important chokepoint? Is it worth pushing with rams/mangonels/petards? Or does one go to imperial yourself and just put your base away from their that you can attack properly? Is there something I'm not getting?