The robot arm is an ongoing computer vision project for which many enhancements have been done during the last years, The aim of the paper is to present a system for controlling a robot arm, based on image processing and recognition. In the frame of this system, an important stage is the calibration. The presentation is focused on a new self-calibration algorithm based on video motion and color detection. The algorithm proves to be general enough to be employed in other similar computer vision scenarios where self-calibration might be useful. Calibration gets performed in a real-time having a low complexity per processed frame of O(n*log(n))