In this paper is shown development of open source software for computer assisted intervention systems. Software libraries are written in C++, but are also accessible from Python, which provides a convenient environment for rapid prototyping and interactive testing. The real-time support includes a device interface and a task library. This paper describes a set of libraries, the Cisst libraries, developed at the Johns Hopkins University to address some of the problems encountered when integrating devices for CAI.