How do I react to a well-placed defensive castle in a 1 TC push?

Because I feel like every game I lose at the moment follows this pattern. I like to be aggressive and I do a load of damage in Feudal and early Castle, they rush up a castle to protect their eco and boom up to Imp, my eco is way behind at this point and I haven't manage a killing blow before the Castle went up (even if I've killed 10 or 15 vils, that's usually balanced out by their boom anyway). If I kill them before their Castle goes up, I win. If they take it to late Castle and Imp, I often lose.

I've tried a few things. I've tried just going round, but usually they're double housewalled, towered, covered elsewhere (or they put up a second and third castle, I feel like half my games end with the opposition having their entire base covered by five castles!), I've tried going for Feudal rams but they're really hard to keep up, especially with repair vils, unless I've got a good pike civ and they don't have infantry. I've tried saying, okay fuck it, I guess I'll go Imp too, but I always hit it way later due to worse eco and then heeeeeeeeere come the trebs.

If the answer is just "don't go 1 TC or get better at winning fast" then...I don't like that answer. I'd rather there was another one before that. But if that's the case, am I undercommitting? Do I want to be diving under TCs more and being more aggressive with forward buildings? I also need to improve my Imp play (I'm not good at manipulating big armies so I tend to lose treb wars and not push effectively) but that's a whole other story.

Cheers!