Learning from experience instead of curated datasets · Oak Lab

Oak Lab

Learning from experience is different from learning from curated datasets. Algorithms that learn from curated datasets can assume that all data is useful for learning. This is usually guaranteed by humans, who collect, clean, and filter raw data so that it is ready to be consumed by a learning algorithm. Experience, on the other hand, does not always contain learnable associations.

In most current applications of deep learning, our systems learn from human-curated datasets, and there has been little incentive for algorithm designers to develop algorithms that can learn from an unfiltered noisy data stream. The inability of existing algorithms to learn from experience is easy to demonstrate empirically with simple examples.

Imagine a simple data stream for learning a supervised prediction task. At every step, the agent observes a single binary-valued feature. If this feature has a value of one, then the ground-truth target is one. Otherwise, the ground-truth target is zero. The value of the feature is sampled from a Bernoulli distribution at every step; the probability of the feature being one is 1% at every time step.

We can visualize the input and target data stream for this task by looking at the values of the feature and target as a function of time. This is shown in Figure 1.

Figure 1. Value of the feature and the ground truth target as a function of time. The target has the same value as the feature.

A linear system can learn this prediction perfectly by learning a weight of one from the feature to the prediction. A simple mechanism to learn this weight is to update the weight parameter with a small step-size parameter in the direction of the gradient computed with respect to the sample mean squared prediction error.

We can make this data stream more complex by adding some noise to the targets. Let's add either +1 or -1, randomly chosen, to the target at every step with a probability of 1%. The updated data stream is shown in Figure 2. Note that only a subset of the targets are predictable in this data stream—namely those that occur when the feature is one. The other +1 and -1 targets are unpredictable and thus unlearnable.

Figure 2. Input and target data stream with ±1 target noise added at each step with 1% probability. Only targets occurring when the feature is one are predictable.

The second data stream is closer to experience than the first, because unfiltered experience has both predictable and unpredictable components. We can make the target more complex by adding noise to it. Let's add noise sampled from a normal distribution with zero mean and a variance of five (a different choice would give similar qualitative results). The targets for this updated data stream are shown in Figure 3. If we look at the targets now, it's no longer easy to find the predictable component from the noise by visual inspection.

Figure 3. Targets with Gaussian noise (zero mean, variance five). The predictable component is no longer visible by inspection.

So far, we have added noise to the targets. In raw experience, the inputs are also messier than our current single-feature input. Let's add 4095 more features that are part of the input data stream. Each of these features also samples its value from a Bernoulli distribution and has a 1% probability of being one. The difference between these features and the earlier feature is that these are uncorrelated with the target and useless for learning.

Now we can learn a linear predictor on this set of features using the noisy targets from Figure 3. Each feature has its own weight parameter connected to the prediction. All weights are initialized to zero. We learn using gradient descent combined with the SGD optimizer. The resulting predictions are shown in Figure 4.

Figure 4. Predictions from a linear model trained with SGD on 4096 features and noisy targets. SGD absorbs the noise in the weight parameters rather than learning only the predictable component.

A key observation to make here is that even though the only learnable aspect of the target is the +1 that occurs when the first feature has a value of one, this is not what SGD learns. SGD and its variants—such as Adam (Kingma & Ba, 2015) and RMSProp (Tieleman & Hinton, 2012)—try to minimize error on all targets and end up absorbing the noise. The reason for this is well understood: SGD assigns the credit for all prediction errors to all parameters with non-zero gradients, even if some of the parameters are connected to signals that are not useful for learning. It has no mechanism to differentiate targets that are predictable from targets that are not predictable and no mechanism to distribute the credit of an error to parameters more selectively.

We now switch to a different algorithm, called Incremental Delta Bar Delta, or IDBD for short (Sutton, 1992). Unlike SGD, IDBD can learn to assign a different amount of credit for a...