It's been a while since we in RelEng started thinking about offloading builds and tests to AWS VMs. Last year we started building Firefox (Linux and Android), Thunderbird and Firefox OS on CentOS 6.2 based AWS VMs. Since then our wait times have been always above 95%, usually around 99%.
However, the story of tests' wait times is different. Since RelEng started building faster, added new products (especially Firefox OS) and more branches, the wait times went below 50%.
It took more than a month to get new platform up and running, properly pupptized and documented. I really liked using mind maps to organize chaotic thoughts, git-buildpackage to keep package building process under control and Upstart for its ability to chain services on system boot.
Chris posted a great overview of what we have now.
I would also like to say THANKS to Armen and Joel for their help with getting tests running on the new platforms, Callek and Dustin for their patience reviewing HUGE patches to get the platform puppetized.