You can semi-DIY it already with Rhasspy + your choice of home automation stack (HomeAssistant or what-have-you). Rhasspy is probably what you're looking for.

https://github.com/rhasspy/rhasspy

Was discussed last year: https://news.ycombinator.com/item?id=33705938

My only real issue in setting it up and running it has been finding appropriate microphones.