I recently received four 24-core Sun X4450 with 128GB RAM, and a Sun X4500 storage box with 8 cores, 32GB RAM, and 24 terabytes of disk space. It took literally less than 2 hours to install Ubuntu 8.0.4 LTS server Linux on all the 4450's (about 15 minute each), but several more days to make some sort of transition over to them.
The machines are fast. Most tests I try run twice as fast on the new machines as on the old sage.math (which was a 1.8Ghz Operton).
The first frustrating transition was that I can no longer run Magma on the new boxes, because they have a different MAC address. I've received one user complaint so far, and expect others, though of course I hope people consider using a viable open source alternative.
I installed about 20 operating systems, including most major Linux distros, FreeBSD, and Solaris under vmware server on one of the boxes, and configured them to all use NFS. Unfortunately, we don't have an NIS/NIS+ or OpenLDAP server setup, so only I can login to those boxes :-(. Setting up NIS or OpenLDAP clients and servers is not for the faint of heart. Anyway, just installing that many Linuxes and configuring them with build tools and NFS took about two days, including time to download installation media. Of all the OS's the only one I gave up on was gentoo, which is a complete disaster (hours of tedious instructions that don't quite work; the live installer crashing; getting the live installer to work, but having vi crash on compilation once it boots up, etc.); that is the worst major operating system I have tried. According to distrowatch, "Gentoo Linux has lost much of its original glory in recent years. Some Gentoo users have come to a realisation that the time-consuming compiling of software packages brings only marginal speed and optimisation benefits. Ever since the resignation of Gentoo's founder and benevolent dictator from the project in 2004, the newly established Gentoo Foundation has been battling with lack of clear directions and frequent developer conflicts..." The failure of Gentoo hopefully helps emphasize the importance of clear direction.
Anyway, it is very nice having about two dozen cleanly installed OS's with a shared filesystem, all running on very fast hardware.
We still haven't installed an operating system on the 24 terabyte disk box, and have a lot of configuration work to do. Fortunately, Tom Boothby is working for me as a sysadmin, so he will do a lot of the hard work.