Horses also take up a lot of time (or rather if you have one, you will spend a lot of time with it to get the most out of what you are paying for it), which means you aren't spending money on other things.
People often make the mistake of thinking "how can they afford <expensive thing> when I can't (because of all the <expensive things> I have to pay for)" without thinking that maybe the other person has a different lifestyle/priories to them.
Horses are expensive to maintain, even if they are gifted to you for free. But people use all sorts of basis for gossip.