I've played with Tree Style Tabs, but it seems like there's some configuration required for the optimal result. So I ask: have you done any configuration changes to it?
I edited userStyles.css to remove the browser tabs from the top of the browser so that only the tst tabs are visible, plus in the tst preferences I added some css to highlight certain tabs such as email and slack. I also added some styles for suspended tabs and removed the a scrollbars.
Also a TST user. It take some getting used to, I don't think I'll ever go back to using horizontal tabs.
It is very much functional out-of-the box and it comes with a lot of preferences to choose from. However for a cleaner interface you'll have to customize your UserChrome.css.
I change the theme and the opening tree behavior do it doesn't nest endlessly. I think I also put to open new tabs at the front of the children, so it is immediately preceding the parent. Just use it for a bit, the top tab bar is so silly.