https://github.com/GPflow/GPflow
Revision b41d4f38436e4a090c940dbd3bc7e2afd39a283e authored by st-- on 23 April 2020, 18:17:42 UTC, committed by GitHub on 23 April 2020, 18:17:42 UTC
Previously, GPflow's NaturalGradient optimizer would call the loss_function once for each (q_mu, q_sqrt) set in the var_list. This is a light refactor that separates out applying the natural gradient step from computing the gradients (`_natgrad_apply_gradients`), and changes `_natgrad_steps` to only evaluate the loss function once, computing the gradients for all (q_mu, q_sqrt) tuples passed in the var_list.

Other changes:
- The no-longer-used `_natgrad_step` method got removed.
- NaturalGradient now takes a `xi_transform` argument that is used for all parameter sets without explicitly specified xi transform (i.e. tuples rather than triplets).
- XiTransform has been changed to have staticmethods.

None of this should affect any downstream code; this PR is backwards-compatible.
1 parent c7550ce
Raw File
Tip revision: b41d4f38436e4a090c940dbd3bc7e2afd39a283e authored by st-- on 23 April 2020, 18:17:42 UTC
refactor natgrads to be more efficient (#1443)
Tip revision: b41d4f3
probability_distributions.py
# Copyright 2017 the GPflow authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Eventually, it would be nice to not have to have our own classes for
# proability distributions. The TensorFlow "distributions" framework would
# be a good replacement.
from .base import TensorType


class ProbabilityDistribution:
    """
    This is the base class for a probability distributions,
    over which we take the expectations in the expectations framework.
    """


class Gaussian(ProbabilityDistribution):
    def __init__(self, mu: TensorType, cov: TensorType):
        self.mu = mu  # [N, D]
        self.cov = cov  # [N, D, D]


class DiagonalGaussian(ProbabilityDistribution):
    def __init__(self, mu: TensorType, cov: TensorType):
        self.mu = mu  # [N, D]
        self.cov = cov  # [N, D]


class MarkovGaussian(ProbabilityDistribution):
    """
    Gaussian distribution with Markov structure.
    Only covariances and covariances between t and t+1 need to be
    parameterised. We use the solution proposed by Carl Rasmussen, i.e. to
    represent
    Var[x_t] = cov[x_t, :, :] * cov[x_t, :, :].T
    Cov[x_t, x_{t+1}] = cov[t, :, :] * cov[t+1, :, :]
    """

    def __init__(self, mu: TensorType, cov: TensorType):
        self.mu = mu  # N+[1, D]
        self.cov = cov  # 2 x (N+1)[, D, D]
back to top