When should you go to stone for a castle drop?

Let’s say you are playing arena and want to FC and drop a castle on the enemies walls. When, and with how many villagers should you gather stone with a regular civ?