Publications by Michael Felsberg

The Fourier descriptors (FDs) is a classical but still popular method for contour matching. The key idea is to apply the Fourier transform to a periodic representation of the contour, which results in a shape descriptor in the frequency domain. FDs are most commonly used to compare object silhouettes and object contours; the authors instead use this well-established machinery to describe local regions to be used in an object-recognition framework. Many approaches to matching FDs are based on the magnitude of each FD component, thus ignoring the information contained in the phase. Keeping the phase information requires us to take into account the global rotation of the contour and shifting of the contour samples. The authors show that the sum-of-squared differences of FDs can be computed without explicitly de-rotating the contours. The authors compare correlation-based matching against affine-invariant Fourier descriptors (AFDs) and WARP-matched FDs and demonstrate that correlation-based approach outperforms AFDs and WARP on real data. As a practical application the authors demonstrate the proposed correlation-based matching on a road sign recognition task.

The major goal of the COSPAL project is to develop an artificial cognitive system architecture, with the ability to autonomously extend its capabilities. Exploratory learning is one strategy that allows an extension of competences as provided by the environment of the system. Whereas classical learning methods aim at best for a parametric generalization, i.e., concluding from a number of samples of a problem class to the problem class itself, exploration aims at applying acquired competences to a new problem class, and to apply generalization on a conceptual level, resulting in new models. Incremental or online learning is a crucial requirement to perform exploratory learning. In the COSPAL project, we mainly investigate reinforcement-type learning methods for exploratory learning, and in this paper we focus on the organization of cognitive systems for efficient operation. Learning is used over the entire system. It is organized in the form of four nested loops, where the outermost loop reflects the user-reinforcement-feedback loop, the intermediate two loops switch between different solution modes at symbolic respectively sub-symbolic level, and the innermost loop performs the acquired competences in terms of perception-action cycles. We present a system diagram which explains this process in more detail. We discuss the learning strategy in terms of learning scenarios provided by the user. This interaction between user (teacher) and system is a major difference to classical robotics systems, where the system designer places his world model into the system. We believe that this is the key to extendable robust system behavior and successful interaction of humans and artificial cognitive systems. We furthermore address the issue of bootstrapping the system, and, in particular, the visual recognition module. We give some more in-depth details about our recognition method and how feedback from higher levels is implemented. The described system is however work in progress and no final results are available yet. The available preliminary results that we have achieved so far, clearly point towards a successful proof of the architecture concept.

In this paper, we present a visual servoing method based on a learned mapping between feature space and control space. Using a suitable recognition algorithm, we present and evaluate a complete method that simultaneously learns the appearance and control of a low-cost robotic arm. The recognition part is trained using an action precedes perception approach. The novelty of this paper, apart from the visual servoing method per se, is the combination of visual servoing with gripper recognition. We show that we can achieve high precision positioning without knowing in advance what the robotic arm looks like or how it is controlled.

In this paper we propose a new approach to real-time view-based pose recognition and interpolation. Pose recognition is particularly useful for identifying camera views in databases, video sequences, video streams, and live recordings. All of these applications require a fast pose recognition process, in many cases video real-time. It should further be possible to extend the database with new material, i.e., to update the recognition system online. The method that we propose is based on P-channels, a special kind of information representation which combines advantages of histograms and local linear models. Our approach is motivated by its similarity to information representation in biological systems but its main advantage is its robustness against common distortions such as clutter and occlusion. The recognition algorithm consists of three steps: (1) low-level image features for color and local orientation are extracted in each point of the image; (2) these features are encoded into P-channels by combining similar features within local image regions; (3) the query P-channels are compared to a set of prototype P-channels in a database using a least-squares approach. The algorithm is applied in two scene registration experiments with fisheye camera data, one for pose interpolation from synthetic images and one for finding the nearest view in a set of real images. The method compares favorable to SIFT-based methods, in particular concerning interpolation. The method can be used for initializing pose-tracking systems, either when starting the tracking or when the tracking has failed and the system needs to re-initialize. Due to its real-time performance, the method can also be embedded directly into the tracking system, allowing a sensor fusion unit choosing dynamically between the frame-by-frame tracking and the pose recognition.

In this paper, we present a new and efficient method to implement robust smoothing of low-level signal features: B-spline channel smoothing. This method consists of three steps: encoding of the signal features into channels, averaging of the channels, and decoding of the channels. We show that linear smoothing of channels is equivalent to robust smoothing of the signal features if we make use of quadratic B-splines to generate the channels. The linear decoding from B-spline channels allows the derivation of a robust error norm, which is very similar to Tukey's biweight error norm. We compare channel smoothing with three other robust smoothing techniques: nonlinear diffusion, bilateral filtering, and mean-shift filtering, both theoretically and on a 2D orientation-data smoothing task. Channel smoothing is found to be superior in four respects: It has a lower computational complexity, it is easy to implement, it chooses the global minimum error instead of the nearest local minimum, and it can also be used on nonlinear spaces, such as orientation space. © 2006 IEEE.

We introduce a compact coding of image information in terms of local multi-modal image descriptors. This coding allows for an explicit separation of the local image information into different visual sub-modalities: geometric information (orientation) and structural image information (contrast transition and colour). Based on this image representation, we derive a similarity function that compares visual information in each of these sub-modalities. This allows for an investigation of the importance of the different factors for stereo matching on a large data set. From this investigation we conclude that it is the combination of visual modalities that gives the best results. Concrete weights for their relative importance are measured. In addition to these quantitative results, we can demonstrate by our simulations that although our image representation reduces image information by 97% we achieve a matching performance which is comparable to block matching techniques. This shows that our very condensed representation preserves the relevant visual information. © 2004 Elsevier B.V. All rights reserved.

This paper introduces a two-dimensional generalization of the analytic signal. This novel approach is based on the Riesz transform, which is used instead of the Hilbert transform. The combination of a 2D signal with the Riesz transformed one yields a sophisticated 2D analytic signal, the monogenic signal. The approach is derived analytically from irrotational and solenoidal vector fields. Based on local amplitude and local phase, an appropriate local signal representation is presented which preserves the split of identity, i.e., the invariance – equivariance property of signal decomposition. This is one of the central properties of the 1D analytic signal that decomposes a signal into structural and energetic information. We show that further properties of the analytic signal concerning symmetry, energy, allpass transfer function, and orthogonality are also preserved, and we compare this to the behavior of other approaches for a 2D analytic signal. As a central topic of this paper, a geometric phase interpretation is introduced which is based on the relation between the 1D analytic signal and the 2D monogenic signal established by the Radon transform. Possible applications of this relationship are sketched and references to other applications of the monogenic signal are given. This report is a revised version of the technical report 2009 [7], and therefore supercedes it.

This review article gives an overview on adaptive filtering methods based on channel representations. The framework of channel representations and its relation to density estimation is introduced. The fast and accurate scheme of virtual shift decoding is introduced and applied in several variants of channel smoothing:

•  channel smoothing with alpha-synthesis for improving stability of edge-enhancing filtering

•  orientation adaptive channel smoothing with applications to coherence-enhancing filtering

•  channel smoothing using graph-cuts for improving filtering results at corners

•  channel-coded feature maps (CCFMs) which lead to a significant speed-up of channel averaging

•  CCFM-based smoothing based on optimal parameters derived from a novel uncertainty relation

For each method, an algorithmic description and some examples of results are provided, together with discussions and references of the original papers. Cross connections to other articles in this volume are given where appropriate.

This paper introduces a bundle adjustment (BA) method that obtains accurate structure and motion from rolling shutter (RS) video sequences: RSBA. When a classical BA algorithm processes a rolling shutter video, the resultant camera trajectory is brittle, and complete failures are not uncommon. We exploit the temporal continuity of the camera motion to define residuals of image point trajectories with respect to the camera trajectory. We compare the camera trajectories from RSBA to those from classical BA, and from classical BA on rectified videos. The comparisons are done on real video sequences from an iPhone 4, with ground truth obtained from a global shutter camera, rigidly mounted to the iPhone 4. Compared to classical BA, the rolling shutter model requires just six extra parameters. It also degrades the sparsity of the system Jacobian slightly, but as we demonstrate, the increase in computation time is moderate. Decisive advantages are that RSBA succeeds in cases where competing methods diverge, and consistently produces more accurate results.

The majority of consumer quality cameras sold today have CMOS sensors with rolling shutters. In a rolling shutter camera, images are read out row by row, and thus each row is exposed during a different time interval. A rolling-shutter exposure causes geometric image distortions when either the camera or the scene is moving, and this causes state-of-the-art structure and motion algorithms to fail. We demonstrate a novel method for solving the structure and motion problem for rolling-shutter video. The method relies on exploiting the continuity of the camera motion, both between frames, and across a frame. We demonstrate the effectiveness of our method by controlled experiments on real video sequences. We show, both visually and quantitatively, that our method outperforms standard structure and motion, and is more accurate and efficient than a two-step approach, doing image rectification and structure and motion.

Segmentation is an important preprocessing step in many applications. Compared to colour segmentation, fusion of colour and depth greatly improves the segmentation result. Such a fusion is easy to do by stacking measurements in different value dimensions, but there are better ways. In this paper we perform fusion using the channel representation, and demonstrate how a state-of-the-art segmentation algorithm can be modified to use channel values as inputs. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using the superparamagnetic clustering algorithm. Our experiments show that depth gradients are more useful than depth values for segmentation, and that channel coding both colour and depth gradients makes tuned parameter settings generalise better to novel images.

Convolution kernels are a commonly used tool in computer vision. These kernels are often specified by an ideal frequency response and the actual filter coefficients are obtained by minimizing some weighted distance with respect to the ideal filter. State-of-the-art approaches usually replace the continuous frequency response by a discrete Fourier spectrum with a multitude of samples compared to the kernel size, depending on the smoothness of the ideal filter and the weight function. The number of samples in the Fourier domain grows exponentially with the dimensionality and becomes a bottleneck concerning memory requirements.

In this paper we propose a method that avoids the discretization of the frequency space and makes filter optimization feasible in higher dimensions than the standard approach. The result is no longer depending on the choice of the sampling grid and it remains exact even if the weighting function is singular in the origin. The resulting improper integrals are efficiently computed using Gauss-Jacobi quadrature.

Techniques from the theory of partial differential equations are often used to design filter methods that are locally adapted to the image structure. These techniques are usually used in the investigation of gray-value images. The extension to color images is non-trivial, where the choice of an appropriate color space is crucial. The RGB color space is often used although it is known that the space of human color perception is best described in terms of non-euclidean geometry, which is fundamentally different from the structure of the RGB space. Instead of the standard RGB space, we use a simple color transformation based on the theory of finite groups. It is shown that this transformation reduces the color artifacts originating from the diffusion processes on RGB images. The developed algorithm is evaluated on a set of real-world images, and it is shown that our approach exhibits fewer color artifacts compared to state-of-the-art techniques. Also, our approach preserves details in the image for a larger number of iterations.

We present a new method for matching a region between an input and a query image, based on the P-channel representation of pixel-based image features such as grayscale and color information, local gradient orientation and local spatial coordinates. We introduce the concept of integral P-channels, which conciliates the concepts of P-channel and integral images. Using integral images, the P-channel representation of a given region is extracted with a few arithmetic operations. This enables a fast nearest-neighbor search in all possible target regions. We present extensive experimental results and show that our approach compares favorably to existing methods for region matching such as histograms or region covariance.

Fourier descriptors (FDs) is a classical but still popular method for contour matching. The key idea is to apply the Fourier transform to a periodic representation of the contour, which results in a shape descriptor in the frequency domain. Fourier descriptors have mostly been used to compare object silhouettes and object contours; we instead use this well established machinery to describe local regions to be used in an object recognition framework. We extract local regions using the Maximally Stable Extremal Regions (MSER) detector and represent the external contour by FDs. Many approaches to matching FDs are based on the magnitude of each FD component, thus ignoring the information contained in the phase. Keeping the phase information requires us to take into account the global rotation of the contour and shifting of the contour samples. We show that the sum-of-squared differences of FDs can be computed without explicitly de-rotating the contours. We compare our correlation based matching against affine-invariant Fourier descriptors (AFDs) and demonstrate that our correlation based approach outperforms AFDs on real world data.

This work presents a novel object tracking approach, where the motion model is learned from sets of frame-wise detections with unknown associations. We employ a higher-order Markov model on position space instead of a first-order Markov model on a high-dimensional state-space of object dynamics. Compared to the latter, our approach allows the use of marginal rather than joint distributions, which results in a significant reduction of computation complexity. Densities are represented using a grid-based approach, where the rectangular windows are replaced with estimated smooth Parzen windows sampled at the grid points. This method performs as accurately as particle filter methods with the additional advantage that the prediction and update steps can be learned from empirical data. Our method is compared against standard techniques on image sequences obtained from an RC car following scenario. We show that our approach performs best in most of the sequences. Other potential applications are surveillance from cheap or uncalibrated cameras and image sequence analysis.

Linear scale-space theory is the fundamental building block for many approaches to image processing like pyramids or scale-selection. However, linear smoothing does not preserve image structures very well and thus non-linear techniques are mostly applied for image enhancement. A different perspective is given in the framework of channel-smoothing, where the feature domain is not considered as a linear space, but it is decomposed into local basis functions. One major drawback is the larger memory requirement for this type of representation, which is avoided if the channel representation is subsampled in the spatial domain. This general type of feature representation is called channel-coded feature map (CCFM) in the literature and a special case using linear channels is the SIFT descriptor. For computing CCFMs the spatial resolution and the feature resolution need to be selected. In this paper, we focus on the spatio-featural scale-space from a scale-selection perspective. We propose a coupled scheme for selecting the spatial and the featural scales. The scheme is based on an analysis of lower bounds for the product of uncertainties, which is summarized in a theorem about a spatio-featural uncertainty relation. As a practical application of the derived theory, we reconstruct images from CCFMs with resolutions according to our theory. The results are very similar to the results of non-linear evolution schemes, but our algorithm has the fundamental advantage of being non-iterative. Any level of smoothing can be achieved with about the same computational effort.

The GPU is the main processing unit on a graphics card. A modern GPU typically provides more than ten times the computational power of an ordinary PC processor. This is a result of the high demands for speed and image quality in computer games. This paper investigates the possibility of exploiting this computational power for tracking points in image sequences. Tracking points is used in many computer vision tasks, such as tracking moving objects, structure from motion, face tracking etc. The algorithm was successfully implemented on the GPU and a large speed up was achieved.

In this paper we propose a new operator which combines advantages of monogenic scale-space and Gaussian scale-space, of the monogenic signal and the structure tensor. The gradient energy tensor (GET) defined in this paper is based on Gaussian derivatives up to third order using different scales. These filters are commonly available, separable, and have an optimal uncertainty. The response of this new operator can be used like the monogenic signal to estimate the local amplitude, the local phase, and the local orientation of an image, but it also allows to measure the coherence of image regions as in the case of the structure tensor. Both theoretically and in experiments the new approach compares favourably with existing methods.

The channel representation allows the construction of soft histograms, where peaks can be detected with a much higher accuracy than in regular hard-binned histograms. This is critical in e.g. reducing the number of bins of generalized Hough transform methods. When applying the maximum entropy method to the channel representation, a minimum-information reconstruction of the underlying continuous probability distribution is obtained. The maximum entropy reconstruction is compared to simpler linear methods in some simulated situations. Experimental results show that mode estimation of the maximum entropy reconstruction outperforms the linear methods in terms of quantization error and discrimination threshold. Finding the maximum entropy reconstruction is however computationally more expensive.

In this paper, we combine the well-established technique of Wiener filtering with an efficient method for robust smoothing: channel smoothing. The main parameters to choose in channel smoothing are the number of channels and the averaging filter. Whereas the number of channels has a natural lower bound given by the noise level and should for the sake of speed be as small as possible, the averaging filter is a less obvious choice. Based on the linear behavior of channel smoothing for inlier noise, we derive a Wiener filter applicable for averaging the channels of an image. We show in some experiments that our method compares favorable with established methods.

 A robust mean value is often a good alternative to the standard mean value when dealing with data containing many outliers. An efficient method for samples of one-dimensional features and the truncated quadratic error norm is presented and compared to the method of channel averaging (soft histograms).

 To program a robot to solve a simple shape-sorter puzzle is trivial. To devise a Cognitive System Architecture, which allows the system to find out by itself how to go about a solution, is less than trivial. The development of such an architecture is one of the aims of the COSPAL project, leading to new techniques in vision based Artificial Cognitive Systems, which allow the development of robust systems for real dynamic environments. The systems developed under the project itself remain however in simplified scenarios, likewise the shape-sorter problem described in the present paper. The key property of the described system is its robustness. Since we apply association strategies of local features, the system behaves robustly under a wide range of distortions, as occlusion, colour and intensity changes. The segmentation step which is applied in many systems known from literature is replaced with local associations and view-based hypothesis validation. The hypotheses used in our system are based on the anticipated state of the visual percepts. This state replaces explicit modeling of shapes. The current state is chosen by a voting system and verified against the true visual percepts. The anticipated state is obtained from the association to the manipulator actions, where reinforcement learning replaces the explicit calculation of actions. These three differences to classical schemes allow the design of a much more generic and flexible system with a high level of robustness. On the technical side, the channel representation of information and associative learning in terms of the channel learning architecture are essential ingredients for the system. It is the properties of locality, smoothness, and non-negativity which make these techniques suitable for this kind of application. The paper gives brief descriptions of how different system parts have been implemented and show some examples from our tests.

In this paper we address the problem of appropriately representing the intrinsic dimensionality of image neighborhoods. This dimensionality describes the degrees of freedom of a local image patch and it gives rise to some of the most often applied corner and edge detectors. It is common to categorize the intrinsic dimensionality (iD) to three distinct cases: i0D, i1D, and i2D. Real images however contain combinations of all three dimensionalities which has to be taken into account by a continuous representation. Based on considerations of the structure tensor, we derive a cone-shaped iD-space which leads to a probabilistic point of view to the estimation of intrinsic dimensionality.

We consider alpha scale spaces, a parameterized class (alpha is an element of (0, 1]) of scale space representations beyond the well-established Gaussian scale space, which are generated by the alpha-th power of the minus Laplace operator on a bounded domain using the Neumann boundary condition. The Neumann boundary condition ensures that there is no grey-value flux through the boundary. Thereby no artificial grey-values from outside the image affect the evolution proces, which is the case for the alpha scale spaces on an unbounded domain. Moreover, the connection between the a scale spaces which is not trivial in the unbounded domain case, becomes straightforward: The generator of the Gaussian semigroup extends to a compact, self-adjoint operator on the Hilbert space L-2(Omega) and therefore it has a complete countable set of eigen functions. Taking the alpha-th power of the Gaussian generator simply boils down to taking the alpha-th power of the corresponding eigenvalues. Consequently, all alpha scale spaces have exactly the same eigen-modes and can be implemented simultaneously as scale dependent Fourier series. The only difference between them is the (relative) contribution of each eigen-mode to the evolution proces. By introducing the notion of (non-dimensional) relative scale in each a scale space, we are able to compare the various alpha scale spaces. The case alpha = 0.5, where the generator equals the square root of the minus Laplace operator leads to Poisson scale space, which is at least as interesting as Gaussian scale space and can be extended to a (Clifford) analytic scale space.

Quadrature filters are a well known method of low-level computer vision for estimating certain properties of the signal, as there are local amplitude and local phase. However, 2D quadrature filters suffer from being not rotation invariant. Furthermore, they do not allow to detect truly 2D features as corners and junctions unless they are combined to form the structure tensor. The present paper deals with a new 2D generalization of quadrature filters which is rotation invariant and allows to analyze intrinsically 2D signals. Hence, the new approach can be considered as the union of properties of quadrature filters and of the structure tensor. The proposed method first estimates the local orientation of the signal which is then used for steering some basis filter responses. Certain linear combination of these filter responses are derived which allow to estimate the local isotropy and two perpendicular phases of the signal. The phase model is based on the assumption of an angular band-limitation in the signal. As an application, a simple and efficient point-of-interest operator is presented and it is compared to the Plessey detector.

Augmented reality is a growing field, with many diverse applications ranging from TV and film production, to industrial maintenance, medicine, education, entertainment and games. The central idea is to add virtual objects into a real scene, either by displaying them in a see-through head-mounted display, or by superimposing them on an image of the scene captured by a camera. Depending on the application, the added objects might be virtual characters in a TV or film production, instructions for repairing a car engine, or a reconstruction of an archaeological site. For the effect to be believable, the virtual objects must appear rigidly fixed to the real world, which requires the accurate measurement in real-time of the position of the camera or the user-s head. Present technology cannot achieve this without resorting to systems that require a significant infrastructure in the operating environment, severely restricting the range of possible applications.

One major goal of the COSPAL project is to develop an artificial cognitive system architecture with the capability of exploratory learning. Exploratory learning is a strategy that allows to apply generalization on a conceptual level, resulting in an extension of competences. Whereas classical learning methods aim at best possible generalization, i.e., concluding from a number of samples of a problem class to the problem class itself, exploration aims at applying acquired competences to a new problem class. Incremental or online learning is an inherent requirement to perform exploratory learning.

Exploratory learning requires new theoretic tools and new algorithms. In the COSPAL project, we mainly investigate reinforcement-type learning methods for exploratory learning and in this paper we focus on its algorithmic aspect. Learning is performed in terms of four nested loops, where the outermost loop reflects the user-reinforcement-feedback loop, the intermediate two loops switch between different solution modes at symbolic respectively sub-symbolic level, and the innermost loop performs the acquired competences in terms of perception-action cycles. We present a system diagram which explains this process in more detail.

We discuss the learning strategy in terms of learning scenarios provided by the user. This interaction between user (’teacher’) and system is a major difference to most existing systems where the system designer places his world model into the system. We believe that this is the key to extendable robust system behavior and successful interaction of humans and artificial cognitive systems.

We furthermore address the issue of bootstrapping the system, and, in particular, the visual recognition module.We give some more in-depth details about our recognition method and how feedback from higher levels is implemented. The described system is however work in progress and no final results are available yet. The available preliminary results that we have achieved so far, clearly point towards a successful proof of the architecture concept.

In this paper we propose a new operator which combines advantages of monogenic scale-space and Gaussian scale-space, of the monogenic signal and the structure tensor. The gradient energy tensor (GET) defined in this paper is based on Gaussian derivatives up to third order using different scales. These filters are commonly available, separable, and have an optimal uncertainty. The response of this new operator can be used like the monogenic signal to estimate the local amplitude, the local phase, and the local orientation of an image, but it also allows to measure the coherence of image regions as in the case of the structure tensor

In this paper we present a new method to implement a robust estimator: B-spline channel smoothing. We show that linear smoothing of channels is equivalent to a robust estimator, where we make use of the channel representation based upon quadratic B-splines. The linear decoding from B-spline channels allows to derive a robust error norm which is very similar to Tukey's biweight error norm. Using channel smoothing instead of iterative robust estimator implementations like non-linear diffusion, bilateral filtering, and mean-shift approaches is advantageous since channel smoothing is faster, it is easy to implement, it chooses the global minimum error instead of the nearest local minimum, and it can also be used on non-linear spaces, such as orientation space. As an application, we implemented orientation smoothing and compared it to the other three approaches.

In this paper we address the problem of appropriately representing the intrinsic dimensionality of image neighborhoods. This dimensionality describes the degrees of freedom of a local image patch and it gives rise to some of the most often applied corner and edge detectors. It is common to categorize the intrinsic dimensionality (iD) to three distinct cases: i0D, i1D, and i2D. Real images however contain combinations of all three dimensionalities which has to be taken into account by a continuous representation. Based on considerations of the structure tensor, we derive a cone-shaped iD-space which leads to a probabilistic point of view to the estimation of intrinsic dimensionality.

In this paper we address the topics of scale-space and phase-based signal processing in a common framework. The involved linear scale-space is no longer based on the Gaussian kernel but on the Poisson kernel. The resulting scale-space representation is directly related to the monogenic signal, a 2D generalization of the analytic signal. Hence, the local phase arises as a natural concept in this framework which results in several advanced relationships that can be used in image processing.