Question about Multiverses.

I realize that the multiverse theory isn't properly proven but I find it difficult to wrap my head around the statement that there are infinitely many universes, given the theory is true. I mean, wouldn't there only be as many universes as there are possible combinations of all atoms in the universe. For simplicity, assume the observable universe was all there was to the universe. That would account for (10^80)! possible arrangements of all atoms and therefore universes (?). Since there is a finite amount of matter in the universe wouldn't an analogous line of thought follow for the entirety of the universe (including things we can't observe-- unless the theory allows for universes with different amounts of matter than ours)?

This is, of course, assuming that single atom differences between universes is enough to distinguish them as two separate universes.

Or is all this just nonsense and there is actual, concrete rigor to the theory that I'm missing? Sorry if this is a dumb question.