The visual perception of independent 3D motion from a moving observer is one of the most challenging tasks in computer vision. This paper presents a powerful fusion of depth and motion information for image sequences. For a large number of points, 3D position and 3D motion is simultaneously estimated by means of Kalman Filters. The necessary ego-motion is computed based on the points that are identified as static points. The result is a real-time system that is able to detect independently moving objects even if the own motion is far from planar. The input provided by this system is suited to be used by high-level perception systems in order to carry out cognitive processes such as autonomous navigation or collision avoidance.