Scaling the Openstack Test Environment

The OpenStack project is developed by one of the largest open-source teams in the world. There are currently over 1000 active contributors world wide producing 1,500 changes per day across more than 250 git projects. Building and testing every change and providing responsive feedback is a huge challenge for us. The way we have addressed the problem is to parallelize all of our builds and tests. Now every change that gets submitted to gerrit triggers multiple builds and tests that run concurrently on multiple OS platforms and even multiple clouds infrastructures. At peak levels the system will run over 20,000 jobs per day. Our system relies on Jenkins CI to run our builds and tests however at that load our CI tools could not keep up with the pace of development so we created a plugin that allows Jenkins to run in a multi-master configuration. Now we can scale our CI tool to contain over 600 build and test nodes connected to 7 Jenkins masters with the added benefit of redundancy to boot.

Experience level: