Sage doesn't really leave a lot of stuff littered about your filesystem. You should first remove your actual Sage folder (in the question you reference, /usr/lib/sagemath/, or for others perhaps in sage-5.12/ in your home directory, depending on where you put it). That will get rid of 99% of things.
The other thing that might be causing trouble is the .sage/ folder, where a lot of prefs and other data is stored (but no functionality). You'd have to remove that folder, which is ordinarily in your home directory, but which can be moved if you use the DOT_SAGE environment variable. The downside is that it will also remove all your Sage notebook worksheets, if you have been using those - it's possible to download them en masse, though, or to save that part of that folder.
Finally, Sage does sometimes create things like .hgrc, which are configuration files for programs like matplotlib and Mercurial that Sage uses. You could remove them, though they should be harmless unless you've done something to them.