(Bug?) Villager standing in foundation

I have been experiencing what seems like a bug which makes the game nearly unplayable for me.

I try to build a building, usually with shift click to put the villager back to work afterwards. But instead of building it the villager moves to the center of the foundation and in case of shift click back to work the villager moves away.

Result: later I look and the villager is standing in the center of the foundation. Or in case of shift click it has moved back to work. Lost countless of games over this.

Is this a known bug? Or do I do something wrong?

It seems to happen mostly in teamgames when there is some lag.