The exclamation point syntax comes from IPython: see their documentation. In particular, in a line starting with !, a single dollar sign indicates a python expression to be expanded, while if you want to pass an actual dollar sign to the shell, use a double dollar sign:
sage: PATH = 33
sage: !echo $PATH 33 sage: !echo$\$PATH

For the second question, you can manipulate the current environment using os.environ, which is a dictionary. For example, os.environ['PATH'] is the current path, which you can modify however you want. I don't know if there are other good ways to modify the path in Python.