How to have villager return to resource automatically after building?

So shift-clicking allows for me to queue up buildings, but it seems like if I shift-click a resource after queuing these then the villager will skip them and head to gathering immediately.

Is there a way to have them return to what they were doing pre-build? Would be super helpful