How to prevent system lock-ups: when a process eats up all the memory

See the linked blog entry.

Except:

More importantly, what strategies could distributions put in place to safeguard from such occurrences? Can this be prevented? Can't the system notice that an application is requesting too much memory?

I think that the most important is to make sure that, for as long as the kernel does not die, a moderately experienced user should be able to recover fairly easily from such a mishap. What is the point of having the mouse pointer being niced up so that it's always responsive, if whatever we click on is not responsive because it is not niced higher than the mad process? Is there a way to have applications in the foreground being automatically niced up so as to respond to user demand?

Another scenario I have encountered many times in the past, back when I had a single-core CPU, is that one process would take 100% of CPU resources, thereby locking the system completely.

The bottom line is that, as long as the linux kernel is still operating fine, a complete lock-up of the system should never occur, never mind how badly some application might behave, or whatever stupid operation a user like me might have attempted.
What can be done to prevent such things? How should the system be configured? What is possible (at least theoretically), and what is not? What solutions already exist??