This dissertation project deals with tracking and re-identification of multiple persons from moving stereo cameras. The scenario it is placed in is a set of moving cars which collaboratively carry out the task: a person or a group of persons, typically walking on a sidewalk of a road or crossing a road at a zebra crossing, is being detected by one stereo system and tracked as long as the persons are visible, resulting in information about existence, number of persons, position, speed, appearance and possibly also their behavior in the immediate future. This information is then transferred to other platforms equipped with stereo sensors, which subsequently need to re-identify and further track the person. Broadcasting the results to nearby cars, these can change their driving characteristics accordingly, which will render traffic more safe.

Detection and tracking should be carried out in a tracking-by-detection scheme in 3D and extend prior work carried out in the Research Training group (Schön et al., 2018), Nguyen et al, 2019) in terms of integration of pedestrian behavior. In this way, not only real-time capability is ensured, but at each epoch during the computations an integrity measure is available for the results. Particular challenge are the constantly changing occlusions coupled with the need for a motion/interaction and behavior prediction model for groups of people.

Another challenge is given by re-identifying problem. It is particularly difficult in outdoors environments and for largely different points of view. In this regard prior work carried out for indoors surveillance, largely based on convolutional neural networks (Blott et al, 2019) is to be extended in a suitable way.


Blott G, Yu J., Heipke C. (2019): Multi-View Person Re-Identification in a Fisheye Camera Network with different viewing directions. PFG, in print.

Nguyen, U.; Rottensteiner, F.; Heipke, C. (2019): Confidence-aware pedestrian tracking using a stereo camera. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences IV-2/W5, 53-60. 10.5194/isprs-annals-IV-2-W5-53-2019

Schön S., Brenner C., Alkhatib H., Coenen M., Dbouk H., Garcia-Fernandez N., Fischer C., Heipke C., Lohmann K., Neumann I., Nguyen U., Paffenholz J.-A., Peters T., Rottensteiner F., Schachtschneider J., Sester M., Sun L., Vogel S., Voges R., Wagner B. (2018). Integrity and Collaboration in Dynamic Sensor Networks. Sensors, 18, 2400, doi:10.3390/s18072400.


Simultaneous localization and mapping (SLAM) is a problem in mobile robotics which requires the robot to create a map of a previously unknown environment and simultaneously localize itself in this map.  Although there exist many approaches to solve this problem, some challenges remain. First, finding correspondences between the map and the robot’s current information often relies on heuristics or requires visual image features. Second, no guarantees can be made for the robot’s position and the correctness of the created map. However, solving the SLAM problem is an important aspect of self-driving cars. Thus, to improve the trust in the correctness of the robot’s information (i.e. integrity), it is desirable to provide guarantees while solving the SLAM problem.

Consequently, we propose to solve the SLAM problem using interval analysis. Interval-based methods are used in the field of mobile robotics to put reliable bounds on the result of different approaches (e.g. localization, map building, obstacle avoidance, etc.) and to guarantee results. A key task in addressing the SLAM problem in the bounded-error context is to associate new information from the robot’s sensors with the existing map. This is also known as the correspondence problem. Moreover, loop closure remains an interesting topic for interval-based methods as Rohou et al. have shown that the closure of a loop can be proven unanimously. Although different sensor combinations are possible to solve the SLAM problem, we propose to first employ a single laser scanner since there exists previous work by Langerwisch and Wagner on the topic of localization OR mapping in a bounded-error context. Afterwards, different sensors (camera, radar, etc.) can be added to further improve the integrity by gathering redundant information.


L. Jaulin, M. Kieffer, O. Didrit, and É. Walter (2001). Applied Interval Analysis. Springer London

M. Langerwisch, B. Wagner (2014). Global Indoor Localization Assuming Unknown But Bounded Sensor Errors. Field and Assistive Robotics - Advances in Systems and Algorithms.

S. Rohou, P. Franek, C. Aubry, L. Jaulin (2018). Proving the existence of loops in robot trajectories. International Journal of Robotics Research.


If integrated systems are able to communicate their decision paths and decision bases to humans, the combined man-machine system becomes less susceptible to errors. The interaction of decisions is conveyed to the human being via visual communication (e.g. by projection into data glasses). The system thus integrates its own position, the position of the other road users (including their predictive behaviour) and communicates this to the user, who can make more reliable decisions on this basis. In the project, the possibilities of visual communication are to be investigated in particular.