What to do when you're a bit behind and start getting towered?

Last two games this has happened to me, where a guy feudals just a bit earlier than me and then behind my tree line he's towering..

So I send a bunch of vils, and they wall/house and repair, and I have to bail, lose a bunch of hitpoints on my vils.

Can't use the wood by my TC now, so gotta make an extra lumber camp, lose 100 wood plus the time.

Then they follow me, push me off that one, repeat.. Now I'm really behind.

Eventually just bail on the whole area, run to safety with as many vils as I can, and everyone hates me because I'm way behind all game.

What am I doing wrong?