Should unique units be available outside castles?

Aside from Goths and Huns (and Sicilians), the main unique unit is only available in castles, limiting their supply and hence gameplay variability between civs. If unique units were available in the respective production building after building a castle (and with a longer creation time than in a castle), this would no longer be the case and they might see more play/there might be more viable strategies. What do you think?

Also, which civ do you think would play most differently with that change?