VAT is a consumption tax. If you consume goods in a place with lower consumption taxes then the tax is doing exactly what it is meant to do.
If you buy consumer goods in a low-VAT place but transport them to a high-VAT place and consume them, then that is clearly illegitimate for the same reason that it is legitimate for VAT to be applied to imports and refunded to exporters.
Within EU, you have free movement of goods, and the VAT is generally paid in the country of sale, not in the country of of the consumer. There are a few exceptions to this, and other regions might have other trade agreements.
Within the EU definition, VAT taxes the company making the sale for the increase in value of a product through the processes the companies and distributors applied to it, and is not a taxation of the receipt of said value.
There is nothing illegitimate about then consuming the product in a different country. It just doesn't happen to send money to your own country's treasury.
If you buy consumer goods in a low-VAT place but transport them to a high-VAT place and consume them, then that is clearly illegitimate for the same reason that it is legitimate for VAT to be applied to imports and refunded to exporters.