refactor: The pytest rework no-one asked for, adjustment to packaging behavior

[tazlin]

Of note:

• comfy_horde.py now imports OK without hordelib.initialise() being called

  • Attempting to instantiate Comfy_Horde will throw a characteristic exception explaining hordelib.initialise() must be called.
    • A new meta test, tests/meta/test_packaging_errors.py, now ensures the behavior is present.
  • There is a dissertation explaining the full rationale for the changes to this file near the top comfy_horde.py.
  • TLDR : Merely navigating an import tree shouldn't be throwing exceptions, even if we are packaging comfy the way we are. This PR adds some inelegant behavior to this module, but it is only as inelegant as doing the packaging in the first place, in my opinion. Further, it at least isolates any suboptimal approaches of this variety to within this module, and without side-effects from leaking out.

• tests/__init__.py now empty; tests/conftest.py now used.

  • The tests now happen in an order which favors fast or problematic tests to be run earlier, and the longer/inference tests to happen later.
  • You can include any fixture in that file in your tests, such as these:

    @pytest.fixture(scope="session")
    def hordelib_instance(init_horde) -> HordeLib:
        return HordeLib()
    @pytest.fixture(scope="class")
    def stable_diffusion_modelname_for_testing(shared_model_manager: type[SharedModelManager]) -> str:
        """Loads the stable diffusion model for testing. This model is used by many tests.
        This fixture returns the model name as string."""
        shared_model_manager.loadModelManagers(compvis=True)
        model_name = "Deliberate"
        assert shared_model_manager.manager.load(model_name)
        return model_name

  Which can be used like so (as seen in test_horde_samplers):

    def test_samplers(
        self,
        stable_diffusion_modelname_for_testing: str,
        hordelib_instance: HordeLib,
    ):

  • Now that Comfy_Horde is importable without an exception being thrown, you can import ShareModelManager for type hinting (or any other purpose).
  • You can omit the type hints, but I would, of course, prefer that they are used.

• Cosine image similarity

  • cv2 (already included upstream/in a dependency) supports image cosine similarity; I have implemented this
  • histogram distance usage/docs/func names clarified as to its purpose
    • While it does work some what on a single machine for determining the probability of two images being similar, it only does so reliably at very low distance scores, on a single machine, and only for certain categories of images. There are a whole class of images which undermine this technique.
  • Cosine similarity is now the primary basis for determining inference result similarity for the purposes of 'expected' images
  • If you would like to compare images in new/existing tests, familiarize yourself with tests/testing_shared_functions.py.
  • All inference tests that previously compared images now do so using at least this technique.
    • By default, many of the inference image similarity tests will not fail on small (or even reasonably substantial) variations in output.
    • LoRa tests are even more lenient (see LORA_SIMILARITY_DEFAULTS)
    • Tests which fail only by virtue of the image not matching the image_expected will generally 'skip' instead of fail. You can see an accounting of passed/error/skipped tests at the end of a tox/pytest run. Passed tests appear as a . from pytest, where a skip is designated with an s instead. Images which are (probably) exceedingly different may still cause a test failure.
  • Did away with IMAGE_DISTANCE_THRESHOLD
  • To configure thresholds or defaults, please see hordelib/utils/distance.py, namely CosineSimilarityResultCode, HistogramDistanceResultCode, INFERENCE_SIMILARITY_DEFAULTS and LORA_SIMILARITY_DEFAULTS

• test_build_helper.py removed, replaced with an attempt to import of build_helper.py

  • See tox.ini for more information.

• Third party reliant code moved from preload.py to comfy_horde.py to better isolate code reliant on packaged libraries.

• Added unloadModelManagers(cls, codeformer: bool = False, compvis: bool = False ...) to SharedModelManagers

• A tiny bit of cleaning up of MM code

[tazlin]

@jug-dev Can you confirm this doesn't introduce any breaking changes to your desired development (code) workflow, when you have the time. I will defer merging until we can agree that there are no systemic changes that will seriously disrupt the way you were developing code.

[jug-dev]

@tazlin Nice. LGTM. No problems with merging this from my point of view.