How to deal with towers being built to disrupt your economy?

I had someone advance to feudal a bit before I did, then before we advanced to castle they just started spamming towers on the periphery of my base.

I didn't have the resources (wood and food) to build sufficient spearmen to take down towers, so I started building my own towers. I also didn't have the resources to age up and every resource nearby was either guarded by a tower or close enough for a tower to attack. Long story short my economy totally collapsed by feudal age.

How do I counter this? Is it just a matter of making sure the enemy can't successfully build these towers or is there a way to actually stop them while they're trying to paralyze your economy?