I build them only once in a while. And I build a recent prerelease only if needed.
Almost all releases, including prereleases, are quite stable. But I usually keep one version of sage, whether prerelease or release, that I know is stable and that I can use for my work.
Any time I have more than two versions installed, I delete the older ones.
I use ccache, and have moved to a system installation of atlas to reduce build times.