Man at arm + thrush how to deal with it?

Hey! I'm a 1850 hd pleb that you can defeat in 10min with some towers. I don't want to be this pleb anymore, that's why I'm looking for answers. :-)

I recently had a 1v1 Arabia game where I got rekt to a strategy I still don't know how to handle properly. I was Teutons and he was Incas. He stole my boar and went 20 pop up Man-at-Arms + tower rush.

I have done some research to find the proper way to handle tower rush. What I found out is:
Scout the enemy a lot. Defensive tower asap and rush his tower down with vills. None of that worked for me that game.

Since he stole my boar, I pushed in 3 deers to make up for the food loss. This delayed my scouting heavily and I had no idea what he was going to do since I didn't scout his base.

He walled up his first tower and had Man-at-arm pushing and defending his villagers.

Even if I put up a defensive tower asap and moved 5 vills to stone, he still had 2-3 towers up before I had the stone to do a second defensive tower. He started to cut off all my resources and I just resigned.

I watched the replay as usual when I lose, and I still don't know exactly what I should have done differently.

What would your strategy be when your opponent stole your boar, into man-at-arm+tower rush?