We introduce a modular matrix factorization framework called "covariate-moderated empirical Bayes matrix factorization" (cEBMF) that can leverage side information to improve the factorization through the use of adaptive priors.