It's not only about the climate but also the rate of change of climate. When the climate changes very quickly, ecosystems cannot adapt and they die.
Remember that humans are actually very adaptable, our range of habitats extends from the arctic to the tropics, desert to rainforest. Most life is not so tolerant.
Indigenous peoples have been living in all corners of the globe for thousands of years. The range of our habitat has stayed roughly the same since antiquity, but the population that any given land can support has been vastly boosted by technology.
Both of these things are true, depending on what is meant by our range of habitat. A small number of people can survive just about anywhere just as you describe, but current population densities are not sustainable in many places without cheap abundant energy.
(I’m not actually concerned with us running out of cheap abundant energy; PV is already fantastic in this regard, and in a hypothetical collapse of all dispatchable power sources that included never being able to make more batteries, we would be severely inconvenienced rather than utterly finished).
