The benefits of continuous test integration are well known. Setting up an infrastruture for this can be a lot of work. It involves integration with version control, automated product building and running of tests, reporting on results in an accessible way, code coverage measuring, and code analysis such as with lint-like tools. We discuss how to set up such an automatic QA system using extensible open source python tools. The focus will be on Buildbot, an open-source, Python-based continuous integration framework. Topics include a survey of the current best available open source software; testing code in languages other than python (and the benefits of using tools written in python in such a situation); strategies for extending these tools to meet custom needs, while avoiding maintainability and quality pitfalls; and special bonus details for QA of projects implemented in python specifically.