Struggling castle age

I usually enter the feudal age in 9-10 minutes, and I usually enter the castle after in 15-17 minutes depending on how I play. The issue I’m having is, right after I hit the castle age my my points drop drastically and I’m like 5th in terms of points. It’s really demoralizing, and I figure the opponent is maybe making troops? But I’m confused on how I can maintain my position at number 1?

Usually just as I enter castle, I always get either double rushed or rushed scout rushed which doesn’t really hurt my economy, but hurts my ability to micromanage. What should I be doing just as I hit castle age? I typically make 3 towncenters and try to boom as fast as possible and that doesn’t even go well.