Added confidence, invariance, consistency metrics (EleutherAI#167)

* Added confidence, invariance, consistency metrics

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Eigenreporter stats, fixed signs

* Update eigen_reporter.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

elk/training/eigen_reporter.py

@@ -107,6 +107,18 @@ def contrastive_xcov(self) -> Tensor:
  def intercluster_cov(self) -> Tensor:
  return self.intercluster_cov_M2 / self.n
 
+ @property
+ def confidence(self) -> Tensor:
+ return self.weight.mT @ self.intercluster_cov @ self.weight
+
+ @property
+ def invariance(self) -> Tensor:
+ return -self.weight.mT @ self.intracluster_cov @ self.weight
+
+ @property
+ def consistency(self) -> Tensor:
+ return -self.weight.mT @ self.contrastive_xcov @ self.weight
+
  def clear(self) -> None:
  """Clear the running statistics of the reporter."""
  self.contrastive_xcov_M2.zero_()