How to not lose holding resources while building a panic Castle

I know it's a simple tip, but I always struggled with this one and maybe it helps someone.

When you finish building a panic castle with villagers that were gathering all kind of resources and you don't know how to re-task them to the right resources, just garrison them into the castle and click on "back to work". They'll return to their previous tasks.