New AoE2 player; How do I improve my castle age time?

I don't seem to lag so much with feudal and imperial age times. But for some reason I always seem to be very late getting to castle. Like playing against AI on moderate I'm always 5-8 mins late on their castle age times. How many farms and villagers do you have when you do you guys make and how age up?

I try to usually always have a vil producing but then getting the 800 food to age seems to take too long. Looking at replays I usually have a higher pop than others before I age up so I guess I'm actually making too many villagers? Has to be either that or not getting enough farms going quick enough?