Herke van Hoof's research while affiliated with University of Amsterdam and other places
What is this page?
This page lists the scientific contributions of an author, who either does not have a ResearchGate profile, or has not yet added these contributions to their profile.
It was automatically created by ResearchGate to create a record of this author's body of work. We create such pages to advance our goal of creating and maintaining the most comprehensive scientific repository possible. In doing so, we process publicly available (personal) data relating to the author as a member of the scientific community.
If you're a ResearchGate member, you can follow this page to keep up with this author's work.
If you are this author, and you don't want us to display this page anymore, please let us know.
It was automatically created by ResearchGate to create a record of this author's body of work. We create such pages to advance our goal of creating and maintaining the most comprehensive scientific repository possible. In doing so, we process publicly available (personal) data relating to the author as a member of the scientific community.
If you're a ResearchGate member, you can follow this page to keep up with this author's work.
If you are this author, and you don't want us to display this page anymore, please let us know.
Publications (64)
Hierarchical methods in reinforcement learning have the potential to reduce the amount of decisions that the agent needs to perform when learning new tasks. However, finding a reusable useful temporal abstractions that facilitate fast learning remains a challenging problem. Recently, several deep learning approaches were proposed to learn such temp...
In reinforcement learning, different reward functions can be equivalent in terms of the optimal policies they induce. A particularly well-known and important example is potential shaping, a class of functions that can be added to any reward function without changing the optimal policy set under arbitrary transition dynamics. Potential shaping is co...
Recent works on machine learning for combinatorial optimization have shown that learning based approaches can outperform heuristic methods in terms of speed and performance. In this paper, we consider the problem of finding an optimal topological order on a directed acyclic graph with focus on the memory minimization problem which arises in compile...
In robotic tasks, we encounter the unique strengths of (1) reinforcement learning (RL) that can handle high-dimensional observations as well as unknown, complex dynamics and (2) planning that can handle sparse and delayed rewards given a dynamics model. Combining these strengths of RL and planning, we propose the Value Refinement Network (VRN), in...
Enabling reinforcement learning (RL) agents to leverage a knowledge base while learning from experience promises to advance RL in knowledge intensive domains. However, it has proven difficult to leverage knowledge that is not manually tailored to the environment. We propose to use the subclass relationships present in open-source knowledge graphs t...
We present Nonparametric Approximation of Inter-Trace returns (NAIT), a Reinforcement Learning algorithm for discrete action, pixel-based environments that is both highly sample and computation efficient. NAIT is a lazy-learning approach with an update that is equivalent to episodic Monte-Carlo on episode completion, but that allows the stable inco...
Reinforcement learning is a promising paradigm for solving sequential decision-making problems, but low data efficiency and weak generalization across tasks are bottlenecks in real-world applications. Model-based meta reinforcement learning addresses these issues by learning dynamics and leveraging knowledge from prior experience. In this paper, we...
Routing problems are a class of combinatorial problems with many practical applications. Recently, end-to-end deep learning methods have been proposed to learn approximate solution heuristics for such problems. In contrast, classical dynamic programming (DP) algorithms guarantee optimal solutions, but scale badly with the problem size. We propose D...
Neural processes (NPs) formulate exchangeable stochastic processes and are promising models for meta learning that do not require gradient updates during the testing phase. However, most NP variants place a strong emphasis on a global latent variable. This weakens the approximation power and restricts the scope of applications using NP variants, es...
Hex is a turn-based two-player connection game with a high branching factor, making the game arbitrarily complex with increasing board sizes. As such, top-performing algorithms for playing Hex rely on accurate evaluation of board positions using neural networks. However, the limited interpretability of neural networks is problematic when the user w...
We consider multi-agent reinforcement learning (MARL) for cooperative communication and coordination tasks. MARL agents can be brittle because they can overfit their training partners' policies. This overfitting can produce agents that adopt policies that act under the expectation that other agents will act in a certain way rather than react to the...
We present Nonparametric Approximation of Inter-Trace returns (NAIT), a Reinforcement Learning algorithm for discrete action, pixel-based environments that is both highly sample and computation efficient. NAIT is a lazy-learning approach with an update that is equivalent to episodic Monte-Carlo on episode completion, but that allows the stable inco...
Enabling reinforcement learning (RL) agents to leverage a knowledge base while learning from experience promises to advance RL in knowledge intensive domains. However, it has proven difficult to leverage knowledge that is not manually tailored to the environment. We propose to use the subclass relationships present in open-source knowledge graphs t...
Mobile health (mHealth) intervention systems can employ adaptive strategies to interact with users. Instead of designing such complex strategies manually, reinforcement learning (RL) can be used to adaptively optimize intervention strategies concerning the user’s context. In this paper, we focus on the issue of overwhelming interactions when learni...
This paper introduces Multi-Agent MDP Homomorphic Networks, a class of networks that allows distributed execution using only local information, yet is able to share experience between global symmetries in the joint state-action space of cooperative multi-agent systems. In cooperative multi-agent systems, complex symmetries arise between different c...
Solving robotic navigation tasks via reinforcement learning (RL) is challenging due to their sparse reward and long decision horizon nature. However, in many navigation tasks, high-level (HL) task representations, like a rough floor plan, are available. Previous work has demonstrated efficient learning by hierarchal approaches consisting of path pl...
Exploration is an essential component of reinforcement learning algorithms, where agents need to learn how to predict and control unknown and often stochastic environments. Reinforcement learning agents depend crucially on exploration to obtain informative data for the learning process as the lack of enough information could hinder effective learni...
Just-in-time adaptive intervention (JITAI) has gained attention recently and previous studies have indicated that it is an effective strategy in the field of mobile healthcare intervention. Identifying the right moment for the intervention is a crucial component. In this paper the reinforcement learning (RL) technique has been used in a smartphone...
Given two sources of evidence about a latent variable, one can combine the information from both by multiplying the likelihoods of each piece of evidence. However, when one or both of the observation models are misspecified, the distributions will conflict. We study this problem in the setting with two conflicting reward functions learned from diff...
Routing problems are a class of combinatorial problems with many practical applications. Recently, end-to-end deep learning methods have been proposed to learn approximate solution heuristics for such problems. In contrast, classical dynamic programming (DP) algorithms can find optimal solutions, but scale badly with the problem size. We propose De...
Reinforcement learning is a promising paradigm for solving sequential decision-making problems, but low data efficiency and weak generalization across tasks are bottlenecks in real-world applications. Model-based meta reinforcement learning addresses these issues by learning dynamics and leveraging knowledge from prior experience. In this paper, we...
We consider the problem of robust multi-agent reinforcement learning (MARL) for cooperative communication and coordination tasks. MARL agents, mainly those trained in a centralized way, can be brittle because they can adopt policies that act under the expectation that other agents will act a certain way rather than react to their actions. Our objec...
In today's clinical practice, magnetic resonance imaging (MRI) is routinely accelerated through subsampling of the associated Fourier domain. Currently, the construction of these subsampling strategies - known as experimental design - relies primarily on heuristics. We propose to learn experimental design strategies for accelerated MRI with policy...
Mobile robot navigation has seen extensive research in the last decades. The aspect of collaboration with robots and humans sharing workspaces will become increasingly important in the future. Therefore, the next generation of mobile robots needs to be socially-compliant to be accepted by their human collaborators. However, a formal definition of c...
Neural processes (NPs) constitute a family of variational approximate models for stochastic processes with promising properties in computational efficiency and uncertainty quantification. These processes use neural networks with latent variable inputs to induce predictive distributions. However, the expressiveness of vanilla NPs is limited as they...
We define hybrid intelligence (HI) as the combination of human
and machine intelligence, augmenting human intellect and
capabilities instead of replacing them and achieving goals
that were unreachable by either humans or machines. HI is an
important new research focus for artificial intelligence, and we
set a research agenda for HI by formulating f...
Neural processes (NPs) constitute a family of vari-ational approximate models for stochastic processes with promising properties in computational efficiency and uncertainty quantification. These processes use neural networks with latent variable inputs to induce predictive distributions. However, the expressiveness of vanilla NPs is limited as they...
Air traffic control is becoming a more and more complex task due to the increasing number of aircraft. Current air traffic control methods are not suitable for managing this increased traffic. Autonomous air traffic control is deemed a promising alternative. In this paper an air traffic control model is presented that guides an arbitrary number of...
Air traffic control is becoming a more and more complex task due to the increasing number of aircraft. Current air traffic control methods are not suitable for managing this increased traffic. Autonomous air traffic control is deemed a promising alternative. In this paper an air traffic control model is presented that guides an arbitrary number of...
This paper introduces MDP homomorphic networks for deep reinforcement learning. MDP homomorphic networks are neural networks that are equivariant under symmetries in the joint state-action space of an MDP. Current approaches to deep reinforcement learning do not usually exploit knowledge about such structure. By building this prior knowledge into p...
Stochastic elements in reinforcement learning (RL) have shown promise to improve exploration and handling of uncertainty, such as the utilization of stochastic weights in NoisyNets and stochastic policies in the maximum entropy RL frameworks. Yet effective and general approaches to include such elements in actor-critic models are still lacking. Ins...
The next generation of mobile robots needs to be socially-compliant to be accepted by humans. As simple as this task may seem, defining compliance formally is not trivial. Yet, classical reinforcement learning (RL) relies upon hard-coded reward signals. In this work, we go beyond this approach and provide the agent with intrinsic motivation using e...
We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement, which reduces variance as it avoids duplicate samples. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators. Combining our estimator with REINFORCE, we obtain a policy gradient estimat...
We present an anytime [1] adaptive sampling technique that generates paths to efficiently measure and then mathematically model a scalar field by performing non-uniform measurements in a given region of interest.
Neural Network based controllers hold enormous potential to learn complex, high-dimensional functions. However, they are prone to overfitting and unwarranted extrapolations. PAC Bayes is a generalized framework which is more resistant to overfitting and that yields performance bounds that hold with arbitrarily high probability even on the unjustifi...
Efficient spatial exploration is a key aspect of search and rescue. In this paper, we present a search algorithm that generates efficient trajectories that optimize the rate at which probability mass is covered by a searcher. This should allow an autonomous vehicle find one or more lost targets as rapidly as possible. We do this by performing non-u...
The well-known Gumbel-Max trick for sampling from a categorical distribution can be extended to sample $k$ elements without replacement. We show how to implicitly apply this 'Gumbel-Top-$k$' trick on a factorized distribution over sequences, allowing to draw exact samples without replacement using a Stochastic Beam Search. Even for exponentially la...
Diversity of environments is a key challenge that causes learned robotic controllers to fail due to the discrepancies between the training and evaluation conditions. Training from demonstrations in various conditions can mitigate---but not completely prevent---such failures. Learned controllers such as neural networks typically do not have a notion...
Building models capable of generating structured output is a key challenge for AI and robotics. While generative models have been explored on many types of data, little work has been done on synthesizing lidar scans, which play a key role in robot mapping and localization. In this work, we show that one can adapt deep generative models for this tas...
In this work, we propose a novel method for training neural networks to perform single-document extractive summarization without heuristically-generated extractive labels. We call our approach BanditSum as it treats extractive summarization as a contextual bandit (CB) problem, where the model receives a document to summarize (the context), and choo...
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and critic. Our algorithm takes the mini...
To learn control policies in unknown environments, learning agents need to explore by trying actions deemed suboptimal. In prior work, such exploration is performed by either perturbing the actions at each time-step independently, or by perturbing policy parameters over an entire episode. Since both of these strategies have certain advantages, a mo...
Learning complex control policies from non-linear and redundant sensory input is an important challenge for reinforcement learning algorithms. Non-parametric methods that approximate values functions or transition models can address this problem, by adapting to the complexity of the data set. Yet, many current non-parametric approaches rely on unst...
Direct contextual policy search methods learn to improve policy parameters and simultaneously generalize these parameters to different context or task variables. However, learning from high-dimensional context variables, such as camera images, is still a prominent problem in many real-world tasks. A naive application of unsupervised dimensionality...
Accurate object shape knowledge provides important information for performing stable grasping and dexterous manipulation. When modeling an object using tactile sensors, touching the object surface at a fixed grid of points can be sample inefficient. In this paper, we present an active touch strategy to efficiently reduce the surface geometry uncert...
For many tasks, tactile or visual feedback is helpful or even crucial. However, designing controllers that take such high-dimensional feedback into account is non-trivial. Therefore, robots should be able to learn tactile skills through trial and error by using reinforcement learning algorithms. The input domain for such tasks, however, might inclu...
Tasks that require many sequential decisions or complex solutions are hard to solve using conventional reinforcement learning algorithms. Based on the semi Markov decision process setting (SMDP) and the option framework, we propose a model which aims to alleviate these concerns. Instead of learning a single monolithic policy, the agent learns a set...
Most manipulation tasks can be decomposed into a sequence of phases, where the robot's actions have different effects in each phase. The robot can perform actions to transition between phases and, thus, alter the effects of its actions, e.g. grasp an object in order to then lift it. The robot can thus reach a phase that affords the desired manipula...
Recognition and manipulation of novel objects in human environments are a prerequisite for many tasks of robots. Since objects often occur in clutter, such robots should be capable of segmenting their environment into individual objects before attempting to learn the objects' properties. In this paper, we propose a probabilistic part-based approach...
Creating robots that can act autonomously in dynamic unstructured environments requires dealing with novel objects. Thus, an offline learning phase is not sufficient for recognizing and manipulating such objects. Rather, an autonomous robot needs to acquire knowledge through its own interaction with its environment, without using heuristics encodin...
Phase transitions in manipulation tasks often occur when contacts between objects are made or broken. A switch of the phase can result in the robot's actions suddenly influencing different aspects of its environment. Therefore, the boundaries between phases often correspond to constraints or subgoals of the manipulation task. In this paper, we inve...
Creating robots that can act autonomously in dynamic, unstructured environments is a major challenge. In such environments, learning to recognize and manipulate novel objects is an important capability. A truly autonomous robot acquires knowledge through interaction with its environment without using heuristics or prior information encoding human d...
Citations
... Early approaches to combat sample inefficiency provide an agent with explicitly formulated task-dependent prior knowledge [1], [2]. Recently, many approaches resort to model-based RL [3]- [6]. A different perspective of describing This work has been financially supported by AI4DI project, which has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 826060. ...
![[object Object]](http://web-wp.archive.org/web/20230120174934im_/https://i1.rgstatic.net/ii/profile.image/1166483711369218-1655122797046_Q64/Qi-Wang-305.jpg)
... The authors of study [48] suggested Deep Policy Dynamic Programming (DPDP), which intends to combine the benefits of established neural heuristics and DP techniques. DPDP prioritizes and restricts the DP state space based on a policy established from a deep neural network trained to estimate edges based on sample solutions. ...
![[object Object]](http://web-wp.archive.org/web/20230120174934im_/https://i1.rgstatic.net/ii/profile.image/913467506692096-1594799033529_Q64/Max-Welling-2.jpg)
... P ATH planning is a method to find an optimal route from the starting point to the target point. It has been widely used in various fields such as robotics [1], [2], [3], drone [4], [5], [6], [7], [8], [9], military service [10], [11], and self-driving car [12], [13]. Recently, reinforcement learning (RL) has been mainly studied for the path planning [3], [7], [9], [10], [11], [14], [15], [16], [17]. ...
... This can be achieved by adaptive learning of dynamic health changes in a patient (Nahum-Shani et al., 2017). Wang, Zhang, et al. (2021) adopted reinforcement learning in a data-driven approach for mobile healthcare user and optimize intervention strategies in their context. Similarly, Gönül learning mechanism to personalize digital adaptive interventions as mobile notifications to the user in coping their health problems. ...
![[object Object]](http://web-wp.archive.org/web/20230120174934im_/https://i1.rgstatic.net/ii/profile.image/279666129686531-1443689009877_Q64/Shihan-Wang-2.jpg)
... Also, the practical usage of our approach should be further evaluated in trials with real users. We have conducted a small-scale feasibility study [34]. Based on the initial results and learned lessons, we plan a longer study to evaluate the effectiveness of our pre-learned delivery strategy for comparable user groups. ...
![[object Object]](http://web-wp.archive.org/web/20230120174934im_/https://i1.rgstatic.net/ii/profile.image/912766672064513-1594631941665_Q64/Monique-Simons-2.jpg)
... Many studies proposed to build semi-synthetic simulators, where the synthetic part is as limited as possible in order to adhere to real-world scenarios. This can for instance be done by using real item embeddings [Shi et al., 2019] or by extending the implicit feedback to unseen data, with debiasing in the missing-not-at-random case [Huang et al., 2020]. Moreover, it is possible to assess the generalizability of a method by benchmarking it against a wide range of simulated configurations, so as to mitigate the influence of simulator design on the results. ...
... For this, the design of robots is required to take human appearances into consideration. In addition, since the navigation tasks are basic and applicable to areas involving humans, social navigation has attracted a lot of researchers [1], [2], [3]. According to Kruse et al. [1], the studies in the field of social navigation have tried to make interaction more comfortable for humans, and there are broadly three goals: comfort, naturalness, and sociability. ...
![[object Object]](http://web-wp.archive.org/web/20230120174934im_/https://i1.rgstatic.net/ii/profile.image/611680312627201-1522847359662_Q64/Florian-Mirus.jpg)
... As artificial intelligence (AI) technologies are playing key roles in our daily lives, developing intelligent systems which can work with humans more effectively (instead of replacing them) is becoming a central research theme [1,2,3]. Such theme is mostly pronounced as hybrid intelligence, aiming to benefit from the strengths of both humans and the machine intelligence in solving problems. ...
... Commonly it is classified as epistemic (model) and aleatoric (data) uncertainty (Der Kiureghian and Ditlevsen 2009;Kendall and Gal 2017). Alternatively, on the basis of input data domain, it can also be classified into in-domain (ID) (Ashukha et al. 2019) and out-ofdomain (OOD) uncertainty (Hendrycks and Gimpel 2017;Wang and Van Hoof 2020). With in-domain data, i.e. the input data distribution is similar to training data distribution, a reliable model should exhibit high predictive performance (e.g., high accuracy or F1-score) and report high confidence (low uncertainty) on correct predictions. ...
... The SAC algorithm has been shown to produce good results when controlling aircraft's trajectory, and safekeeping a minimum distance from obstacles or other aircraft [4], [5]. Additionally, it has been proven that an RL agent can simultaneously control vertical and lateral actions [6]. Finally, previous work also suggests that RL agents can handle uncertainties regarding the position of other aircraft [7], [8]. ...
![[object Object]](http://web-wp.archive.org/web/20230120174934im_/https://i1.rgstatic.net/ii/profile.image/908798092922880-1593685758254_Q64/Joris-Mollinga.jpg)
