Counter vs Franks fast castle

Hi guys,

I am relatively new to AoE2 and have round about 50 games and I am slightly above 900 elo at the moment.

Recently I thought I was tower rushed by a Franks-Player. He just placed 1 tower close to my wall that included a lot of buildings. I did not really worry and concentrated on macro and did a little damage with scouts in Feudal age.

But at the beginning of castle age he dropped a castle within range of his tower, supported by his whole army.

1. Am I supposed to prevent this by pulling a lot of vills and my whole army, although I would send them within the range of the tower?
2. When the castle is established (map was small), what is my gameplan right now? He quickly built axethrowers so I had to defend and felt stuck...

Thx in advance!