Falling off in Castle Age

I'm really new to the game having picked it up about a week ago after only having played as a child with cheats on. I have been watching a lot of videos and matches. I'm only playing with the AI on intermediate and I am able to keep up with them in score until I hit the Castle Age and then they pull ahead. I know of fast Castle, but the two times I tried it I got over run by ground units and didn't have an army because I was focused on booming. I think it may be because my Fast Castle is really slow. What are some simple things I may be doing wrong to fall behind? Other than improving my fast castle because atm I can only score a D on that fast castle tutorial.

Thanks in advance.