Lost a game because villager goes to the outside of wall to build

Hi!

I play around 900 elo. I tasked vills to wall up my base in the beginning of the game and as sometimes happens one vill started building outside of the wall. I did not notice this until the wall finished, so I tasked the villager to build a gate in the wall. At some point later I pressed idle vill and tasked it to build a farm. At that very moment an enemy m@a rush shows up and passes through the gate simultaneously as the vill. I was delayed by the rush and got smashed in imp.

I realise I could have queued up a task for my vill to go inside the base, however that misses the point that I don't seem to be in control of which side villagers build walls from. How come this is the case? How do people deal with this?