When fitting a mixture of Gaussians to training data there are usually two choices for the type of Gaussians used. Either diagonal or full covariance. Imposing a structure, though may be restrictive and lead to degraded performance and/or increased computations. In this work we use several criteria to estimate the structure of regression matrices of a mixture of Gaussians. Unlike many previous approaches, the criteria tested in this work attempt to estimate a discriminative structure, which is suited for classification tasks. We report results on the 1996 NIST speaker recognition task and compare the performance of our criteria to non-discriminative strtucture-finding algorithm, like structural EM.