I'm a doctor-in-training considering a second career in physics or engineering. How stupid am I?

Hey all,

I'm a 29-year-old MD. I was a physics major in undergrad, and now I'm doing my residency.

I'll keep this brief. After I graduate from residency, I am likely to make >$200k per year. I currently owe about $500k (I have loans from both undergrad and med school).

I chose to do medicine because I thought that the human interactions with patients would be intrinsically fulfilling. And they are - to an extent. But as I get further in clinical medicine, I am discovering how much I hate the interrupted and punctuated schedule of seeing patients. I realize that I want to sit at my desk and just work deeply on interesting problems - I don't want to constantly be getting paged that my next patient is here. I miss the deep problem solving of physics.

I have this idle fantasy where I work my ass off for 4-5 years and pay off my debt/make it manageable, then go to physics grad school.

Is this a stupid idea? Is there anybody here who has done physics as a second career? What are your thoughts?