What to do when archer rushed

I played a game today, where the enemy archer rushed me. I saw his range and archers before they got to me, so I made a range and made defensive skirms. (Meanwhile I was scout rushing him, which was really just my scouts against some archers). Was trying to click up to castle soon. Should I have kept making skirms, or made castle my priority? Then when I got to castle, I started making knights. Should I have? Soon after I made kts, I ran into his cav archers too.