Purpose: There is a need for tools to study real-world communication abilities in people with hearing loss. We outline a potential method for this that analyzes gaze and use it to answer the question of when and how much listeners with hearing loss look toward a new talker in a conversation.Method: Twenty-two older adults with hearing loss followed a prerecorded two person audiovisual conversation in the presence of babble noise. We compared their eye-gaze direction to the conversation in two multilevel logistic regression (MLR) analyses. First, we split the conversation into events classified by the number of active talkers within a turn or a transition, and we tested if these predicted the listener's gaze. Second, we mapped the odds that a listener gazed toward a new talker over time during a conversation transition.Results: We found no evidence that our conversation events predicted changes in the listener's gaze, but the listener's gaze toward the new talker during a silence-transition was predicted by time: The odds of looking at the new talker increased in an s-shaped curve from at least 0.4 s before to 1 s after the onset of the new talker's speech. A comparison of models with different random effects indicated that more variance was explained by differences between individual conversation events than by differences between individual listeners.Conclusions: MLR modeling of eye-gaze during talker transitions is a promising approach to study a listener's perception of realistic conversation. Our experience provides insight to guide future research with this method.

This letter investigates relationships between iterated filtering algorithms based on statistical linearization, such as the iterated unscented Kalman filter (IUKF), and filtering algorithms based on quasi–Newton (QN) methods, such as the QN iterated extended Kalman filter (QN–IEKF). Firstly, it is shown that the IUKF and the iterated posterior linearization filter (IPLF) can be viewed as QN algorithms, by finding a Hessian correction in the QN –IEKF such that the IPLF iterate updates are identical to that of the QN–IEKF. Secondly, it is shown that the IPLF/ IUKF update can be rewritten such that it is approximately identical to the QN–IEKF, albeit for an additional correction term. This enables a richer understanding of the properties of iterated filtering algorithms based on statistical linearization.

The ability to understand speech in complex listening environments reflects an interaction of cognitive and sensory capacities that are difficult to capture with behavioural tests. The study of natural listening behaviours may lead to the development of new metrics that better reflect real-life communication abilities. To this end, we investigated the relationship between speech comprehension and eye-movements among hearing-impaired people in a challenging listening situation. While previous research has investigated the effect of background noise on listeners’ gaze patterns with single talkers, the effect of noise in multitalker conversations remains unknown. We tracked eye-movements of seven aided hearing-impaired adults while they viewed video recordings of two life-sized talkers engaged in an unscripted dialogue. Hearing loss ranged from moderate to severe. We used multiple-choice questions to measure participants’ comprehension of the conversation in multitalker babble noise at three different signal-to-noise ratios. All participants made saccades between the two talkers more frequently than the talkers’ conversational turns. This measure tended to correlate positively with participants’ comprehension scores, but the effect was significant in only one signal-to-noise ratio condition. Post-hoc investigation suggests that intertalker saccade rate is driven by an interaction of hearing ability and conversational turn-taking events, which will be further discussed.

People with hearing impairment typically have difficulties following conversations in multi-talker situations. Previous studies have shown that utilizing eye gaze to steer audio through beamformers could be a solution for those situations. Recent studies have shown that in-ear electrodes that capture electrooculography in the ear (EarEOG) can estimate the eye-gaze relative to the head, when the head was fixed. The head movement can be estimated using motion sensors around the ear to create an estimate of the absolute eye-gaze in the room. In this study, an experiment was designed to mimic a multi-talker situation in order to study and model the EarEOG signal when participants attempted to follow a conversation. Eleven hearing impaired participants were presented speech from the DAT speech corpus (Bo Nielsen et al., 2014), with three targets positioned at -30 degrees, 0 degrees and +30 degrees azimuth. The experiment was run in two setups: one where the participants had their head fixed in a chinrest, and the other where they were free to move their head. The participants task was to focus their visual attention on an LED-indicated target that changed regularly. A model was developed for the relative eye-gaze estimation, taking saccades, fixations, head movement and drift from the electrode-skin half-cell into account. This model explained 90.5% of the variance of the EarEOG when the head was fixed, and 82.6% when the head was free. The absolute eye-gaze was also estimated utilizing that model. When the head was fixed, the estimation of the absolute eye-gaze was reliable. However, due to hardware issues, the estimation of the absolute eye-gaze when the head was free had a variance that was too large to reliably estimate the attended target. Overall, this study demonstrated the potential of estimating absolute eye-gaze using EarEOG and motion sensors around the ear.

Courses at the Master’s level in automatic control and signal processing cover mathematical theories and algorithms for control, estimation, and filtering. However, giving students practical experience in how to use these algorithms is also an important part of these courses. A goal is that the students should not only be able to understand and derive these algorithms, but also be able to apply them to real-life technical problems. The latter is achieved by assigning more time to the laboratory tutorials and designing them in such a way that the exercises are open for interpretation; an example of this would be giving the students more freedom to decide how to acquire the data needed to solve the given exercises.The students are asked to hand in a laboratory report in which they describe how they solved the exercises. This paper presents a double-blind peer-review process for laboratory reports, introduced at the Department of Electrical Engineering, Linköping University, Sweden. A survey was administered to students, and the results are summarized in this paper. Also discussed are the teachers’ experiences of peer review and of how students perform later in their education in writing their Master’s theses.

Hearing-impaired listeners have a reduced ability to selectively attend to sounds of interest amid distracting sounds in everyday environments. This ability is not fully regained with modern hearing technology. A better understanding of the brain mechanisms underlying selective attention during speech processing may lead to brain-controlled hearing aids with improved detection and amplification of the attended speech. Prior work has shown that brain responses to speech, measured with magnetoencephalography (MEG) or electroencephalography (EEG), are modulated by selective attention. These responses can be predicted from the speech signal through linear filters called Temporal Response Functions (TRFs). Unfortunately, these sensor-level predictions are often noisy and do not provide much insight into specific brain source locations. Therefore, a novel method called Neuro-Current Response Functions (NCRFs) was recently introduced to directly estimate linear filters at the brain source level from MEG responses to speech from one talker. However, MEG is not well-suited for wearable and realtime hearing technologies. This work aims to adapt the NCRF method for EEG under more realistic listening environments. EEG data was recorded from a hearing-impaired listener while attending to one of two competing talkers embedded in 16-talker babble noise. Preliminary results indicate that source-localized linear filters can be directly estimated from EEG data in such competing-talker scenarios. Future work will focus on evaluating the current method on a larger dataset and on developing novel methods, which may aid in the improvement of next-generation brain-controlled hearing technology.

Recent studies of selective auditory attention have demonstrated that neural responses recorded with electroencephalogram (EEG) can be decoded to classify the attended talker in everyday multitalker cocktail-party environments. This is generally referred to as the auditory attention decoding (AAD) and could lead to a breakthrough for the next-generation of hearing aids (HAs) to have the ability to be cognitively controlled. The aim of this paper is to investigate whether cepstral analysis can be used as a more robust mapping between speech and EEG. Our preliminary analysis revealed an average AAD accuracy of 96%. Moreover, we observed a significant increase in auditory attention classification accuracies with our approach over the use of traditional AAD methods (7% absolute increase). Overall, our exploratory study could open a new avenue for developing new AAD methods to further advance hearing technology. We recognize that additional research is needed to elucidate the full potential of cepstral analysis for AAD.

In this work, estimators for platform pose and landmark maps for visual-inertial data are analysed. It is shown that the full, non-linear, visual-inertial problem has a conditionally linear substructure in the 2D case which can be exploited for efficient solutions, e.g., Block Coordinate Descent (BCD). It is also shown that the measurement noise from the non-linear model becomes parameter dependent resulting in biased estimates if that fact is ignored. However, the bias can be accounted for using the Iteratively Reweighted Least Squares (IRLS) method. In the 3D case the conditionally linear substructure is not separable. However, it can be shown that the Jacobian of the non-linear substructure can be calculated recursively resulting in an efficient solution. A simulated 2D visual-inertial example is used to illustrate the theoretical results.

The unscented Kalman filter (UKF) is a very popular solution for estimation of the state in nonlinear systems. Similar to the extended Kalman filter (EKF) and contrary to the Kalman filter (KF) for linear systems, the UKF provides no guarantees that the filter updates will improve the filtered state estimate. In the past, the iterated EKF (IEKF) has been suggested as a way to online monitor the filter performance and try to improve it using optimization techniques. In this paper we do the same for the UKF, deriving six iterated UKF (IUKF) variations based on two cost functions and three optimization algorithms. The methods are evaluated and compared to IEKF versions and to two versions of the iterative posterior linearization filter (IPLF) in three benchmark simulation studies. The results show that IUKF algorithms can be used as a derivative free alternative to IEKF, and provide insights about the different design choices available in IUKF algorithms.

This paper presents a batch estimation method for Simultaneous Localization and Mapping (SLAM) using the Prediction Error Method (PEM). The estimation problem considers landmarks as parameter while treating dynamics using state space models. The gradient needed for parameter estimation is computed recursively using an Extended Kalman Filter (EKF). Results using simulations with a monocular camera and inertial sensors are presented and compared to a Nonlinear Least- Squares (NLS) estimator. The presented method produce both lower RMSE’s and scale better to the batch length. 

This paper presents a method for global pose estimation using inertial sensors, monocular vision, and ultra wide band (UWB) sensors. It is demonstrated that the complementary characteristics of these sensors can be exploited to provide improved global pose estimates, without requiring the introduction of any visible infrastructure, such as fiducial markers. Instead, natural landmarks are jointly estimated with the pose of the platform using a simultaneous localization and mapping framework, supported by a small number of easy-to-hide UWB beacons with known positions. The method is evaluated with data from a controlled indoor experiment with high precision ground truth. The results show the benefit of the suggested sensor combination and suggest directions for further work.

This paper presents a system which combines a zero-velocity-update-(ZUPT-)aided inertial navigation system (INS), using a foot-mounted inertial measurement unit (IMU), with opportunistic use of multi-frequency received signal strength (RSS) measurements. The system does not rely on maps or pre-collected data from surveys of the radio-frequency (RF) environment. Instead it builds its own database of collected RSS measurements during the course of the operation. New RSS measurements are continuously compared with the stored values in the database, and when the user returns to a previously visited area this can thus be detected. This enables loop-closures to be detected online and used for error drift correction. The system utilises a distributed particle simultaneous localization and mapping (DP-SLAM) algorithm which provides a flexible 2D navigation platform that can be extended with more sensors. The experimental results presented in this paper indicates that the developed RSS SLAM algorithm can, in many cases, significantly improve the positioning performance of a foot-mounted INS. 

This paper studies the accuracy of the estimated eyepoint of an Optical See-Through Head-Mounted Display (OST HMD) calibrated using the Single Point Active Alignment Method (SPAAM). Quantitative evaluation of calibration procedures for OST HMDs is complicated as it is currently not possible to share the subject’s view. Temporarily replacing the subject’s eye with a camera during the calibration or evaluation stage has been proposed, but the uncertainty of a correct eyepoint estimation remains. In the experiment reported in this paper, subjects were used for all stages of calibration and the results were verified with a 3D measurement device. The nine participants constructed 25 visual alignments per calibration after which the estimated pinhole camera model was decomposed into its intrinsic and extrinsic parameters using two common methods. Unique to this experiment, compared to previous evaluations, is the measurement device used to cup the subject’s eyeball. It measures the eyepoint location relative to the head tracker, thereby establishing the calibration accuracy of the estimated eyepoint location. As the results on accuracy are expressed as individual pinhole camera parameters, rather than a compounded registration error, this paper complements  previously published work on parameter variance as the former denotes bias and the latter represents noise. Results indicate that the calibrated eyepoint is on average 5 cm away from its measured location and exhibits a vertical bias which potentially causes dipvergence for stereoscopic vision for objects located further away than 5.6 m. Lastly, this paper closes with a discussion on the suitability of the traditional pinhole camera model for OST HMD calibration.

The parameter estimation variance of the Single Point Active Alignment Method (SPAAM) is studied through an experiment where 11 subjects are instructed to create alignments using an Optical See-Through Head Mounted Display (OSTHMD) such that three separate correspondence point distributions are acquired. Modeling the OSTHMD and the subject's dominant eye as a pinhole camera, findings show that a correspondence point distribution well distributed along the user's line of sight yields less variant parameter estimates. The estimated eye point location is studied in particular detail. Thefindings of the experiment are complemented with simulated datawhich show that image plane orientation is sensitive to the numberof correspondence points. The simulated data also illustrates someinteresting properties on the numerical stability of the calibrationproblem as a function of alignment noise, number of correspondencepoints, and correspondence point distribution.

Navigation and mapping in unknown environments is an important building block for increased autonomy of unmanned vehicles, since external positioning systems can be susceptible to interference or simply being inaccessible. Navigation and mapping require signal processing of vehicle sensor data to estimate motion relative to the surrounding environment and to simultaneously estimate various properties of the surrounding environment. Physical models of sensors, vehicle motion and external influences are used in conjunction with statistically motivated methods to solve these problems. This thesis mainly addresses three navigation and mapping problems which are described below.

We study how a vessel with known magnetic signature and a sensor network with magnetometers can be used to determine the sensor positions and simultaneously determine the vessel's route in an extended Kalman filter (EKF). This is a so-called simultaneous localisation and mapping (SLAM) problem with a reversed measurement relationship.

Previously determined hydrodynamic models for a remotely operated vehicle (ROV) are used together with the vessel's sensors to improve the navigation performance using an EKF. Data from sea trials is used to evaluate the system and the results show that especially the linear velocity relative to the water can be accurately determined.

The third problem addressed is SLAM with inertial sensors, accelerometers and gyroscopes, and an optical camera contained in a single sensor unit. This problem spans over three publications.

We study how a SLAM estimate, consisting of a point cloud map, the sensor unit's three dimensional trajectory and speed as well as its orientation, can be improved by solving a nonlinear least-squares (NLS) problem. NLS minimisation of the predicted motion error and the predicted point cloud coordinates given all camera measurements is initialised using EKF-SLAM.

We show how NLS-SLAM can be initialised as a sequence of almost uncoupled problems with simple and often linear solutions. It also scales much better to larger data sets than EKF-SLAM. The results obtained using NLS-SLAM are significantly better using the proposed initialisation method than if started from arbitrary points. A SLAM formulation using the expectation maximisation (EM) algorithm is proposed. EM splits the original problem into two simpler problems and solves them iteratively. Here the platform motion is one problem and the landmark map is the other. The first problem is solved using an extended Rauch-Tung-Striebel smoother while the second problem is solved with a quasi-Newton method. The results using EM-SLAM are better than NLS-SLAM both in terms of accuracy and complexity.

Simultaneous Localisation and Mapping (SLAM) denotes the problem of jointly localizing a moving platform and mapping the environment. This work studies the SLAM problem using a combination of inertial sensors, measuring the platform's accelerations and angular velocities, and a monocular camera observing the environment. We formulate the SLAM problem on a nonlinear least squares (NLS) batch form, whose solution provides a smoothed estimate of the motion and map. The NLS problem is highly nonconvex in practice, so a good initial estimate is required. We propose a multi-stage iterative procedure, that utilises the fact that the SLAM problem is linear if the platform's rotations are known. The map is initialised with camera feature detections only, by utilising feature tracking and clustering of  feature tracks. In this way, loop closures are automatically detected. The initialization method and subsequent NLS refinement is demonstrated on both simulated and real data.