Does it matter where you get your undergraduate physics degree?

Hi, I'm an international student in the UK (senior) looking to study in the US. Since I would need to sell a kidney to pay for the top school tuitions, I'm looking at Universities with physics departments that are less well known. One of these is Washington University in St Louis, which I could get good financial aid since my stats are higher than average there, and it's not as popular with internationals than the Ivy Leagues or MIT or Stanford etc.

My issue is whether this would be a detriment in any way. I understand that physics is the same no matter where it's taught, but would to going to a lesser known and much lower ranked physics college affect my undergraduate research and graduate prospects? Would the physics students me pushed to do loads of different and exciting courses, or would the course variety be limiting and stagnant? Have any of you guys been to a lesser known college but ended up successful physicists?

I could go to a top Uni in the UK, but the studying the US has been a dream of mine for a long time, and I don't want this to put me off! I would appreciate any responses, thank you :)