Is there a point in rushing skilled players who wall?

If they wall themselves completely early, then scouts and militia rushes aren't gonna work. Archers have some range but can't impact too much. Maybe I should go for fast castle?