If I can't fix it in like 3 tries, I switch to my local shell script having the script to run on the remote end be in a HEREDOC and just scp it it over and then ssh exec it. Inside the HEREDOC, it's a sane environment.
Copying over the commands to run also has another added benefit, you have quite good documentation of what was run and when. Also it allows one to easily run a failing thing again (in case output gets mangled in the executing script)