Doesn't help in this case but in your /etc/ssh/sshd_config or similar change or, if missing, add

    PubkeyAuthentication yes

    ChallengeResponseAuthentication no
    PasswordAuthentication no
    PermitRootLogin no

Make sure you can log in using your key and su/sudo from that unprivileged user before applying those changes.