Building houses to serve as walls. Is it advisable?

I've been trying to place houses wherever I'd place walls, mostly at choke points. This serves me well as houses are available immediately, as opposed to stone walls which are only available after Feudal Age. They are also stronger than pallisade walls.

However, my villagers spend a lot of time walking over to the build site, and I wonder if this strategy is too inefficient.