Users don't, but I'll bet a lot of hackers do. Apparently it has had an effect on rankings - what you consider to be "little effect" can be huge deal to someone whose business relies heavily on referrals from Google. And the speed difference that I've experienced is actually quite noticeable.
edit: To give some concrete data, I took the queries I've done on Google today and compared the first page of results. Roughly 40% of the results were ranked differently than they were before. News results were more recent with Caffeine. Previous requests took perhaps, 1/2 a second to display - it's relatively instantaneous now.
I am wondering, however, if the display time is faster now because the new site doesn't have much user load where the old google site is relatively busy.
The claim about 'faster' is that newer search results show up sooner than they did on the old google. That is, new results get indexed for display earlier.
edit: To give some concrete data, I took the queries I've done on Google today and compared the first page of results. Roughly 40% of the results were ranked differently than they were before. News results were more recent with Caffeine. Previous requests took perhaps, 1/2 a second to display - it's relatively instantaneous now.