Mathematical consequences of orthogonal weights initialization and regularization in deep learning. Experiments with gain-adjusted orthogonal regularizer on RNNs with SeqMNIST dataset.