In a flat universe, the universe starts off infinitely large and infinitely dense at time 0. As the universe expands, it gets less dense. Imagine an infinitely dense rubber sheet that starts to stretch at time 0, and after time 0 it is no longer infinitely dense, and it keeps slowly stretching forever, getting less and less dense over time. That's the universe as it is currently envisioned.
But our "Hubble sphere" is in no way special compared to other Hubble spheres. Our Hubble sphere is the part of the universe surrounding us from which light has arrived. Outside that sphere, nothing has a causal connection to us, since nothing can travel faster than the speed of light.
I.e., it just wouldn't be the case that our Hubble sphere is full of stuff and all the other Hubble spheres are are so different as to be empty.
But our "Hubble sphere" is in no way special compared to other Hubble spheres. Our Hubble sphere is the part of the universe surrounding us from which light has arrived. Outside that sphere, nothing has a causal connection to us, since nothing can travel faster than the speed of light.
I.e., it just wouldn't be the case that our Hubble sphere is full of stuff and all the other Hubble spheres are are so different as to be empty.
Unless, of course, everything we know is wrong.