Remove mentions to .egg and stop producing them from kedro package

[astrojuanlu]

Description

Fix gh-2273.

In addition, I fixed some low-hanging fruit for gh-2414, but found some complex issues along the way (gh-2542, gh-2567) so I didn't go any further. On the other hand, the micropkg workflow is somewhat self-contained and is not affected by how starters look, so it can be refactored separately. Reverted, see below.

Development notes

Added https://pypi.org/project/build/ as runtime dependency.

Even though in principle we should be ready to move forward with gh-2350, just to avoid any issues I think it will be better to wait until https://github.com/kedro-org/kedro/milestone/36 is complete.

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

[merelcht]

Nice one! 👍 I actually started looking into this as well and completely agree that the micropkg flow should be tackled separately. It's not entirely clear to me yet how to replace the reliance on .egg-info.

[stichbury]

LGTM 🏆

[astrojuanlu]

Hmm this error has happened twice and maybe it's legitimate:

        result = CliRunner().invoke(
            fake_project_cli, ["pipeline", "create", PIPELINE_NAME], obj=fake_metadata
        )
        assert result.exit_code == 0
        result = CliRunner().invoke(
            fake_project_cli,
            ["micropkg", "package", f"pipelines.{PIPELINE_NAME}"] + options,
            obj=fake_metadata,
        )
    
>       assert result.exit_code == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <Result PermissionError(13, 'The process cannot access the file because it is being used by another process')>.exit_code

[astrojuanlu]

In case it's okay for kedro micropkg to drop the concurrency capabilities it seemingly had until now, I'm applying this solution to amend the tests: pytest-dev/pytest-xdist#385 (comment)

[astrojuanlu]

Despite my last commit, 1 of the tests is still failing. More reason to think there is a legitimate problem I'm not seeing. Will need to debug on a Windows machine.

[astrojuanlu]

One interesting thing from the logs: the fake_root_dir fixture is supposed to yield a different temporary directory for each module:

kedro/tests/framework/cli/conftest.py

Lines 46 to 53 in 6239e78

	@fixture(scope="module")
	def fake_root_dir():
	# using tempfile as tmp_path fixture doesn't support module scope
	tmpdir = tempfile.mkdtemp()
	try:
	yield Path(tmpdir).resolve()
	finally:
	shutil.rmtree(tmpdir, ignore_errors=True)

However, the two tests that were failing (before I added @pytest.mark.xdist_group) point to the same directory:

https://app.circleci.com/pipelines/github/kedro-org/kedro/21657/workflows/f45fcaad-c5a2-4e00-8176-2362f6617f5c/jobs/252386/parallel-runs/0/steps/0-117

--outdir 'C:\Users\circleci\AppData\Local\Temp\tmpi8davw_b\dummy_project\dist'

😬 Could do some investigation as to whether this is a problem in pytest, pytest-xdist, or a nasty bug of tempfile.mkdtemp in Python 3.7.12... But it would take some time and I'm not getting into that rabbit hole if I can avoid it.

I tried locally to remove scope="module" but I was getting a bunch of FileNotFoundError coming from various places (cleanup fixtures and chdir_to_dummy_project) so for now I'm going to attempt @merelcht suggestion of removing the assertion on the exit code and cross my fingers...

[astrojuanlu]

Test still failing:

>       assert success_message in result.output
E       assert "'dummy_package.pipelines.my_pipeline' packaged!" in "Creating 'C:\\Users\\circleci\\AppData\\Local\\Temp\\tmp6j4gzjyl\\dummy_package\\pipelines\\my_pipeline\\nodes.py': O...\python.exe' -m build --sdist --outdir 'C:\\Users\\circleci\\AppData\\Local\\Temp\\tmpy65f9iu2\\dummy_project\\dist'\n"
E        +  where "Creating 'C:\\Users\\circleci\\AppData\\Local\\Temp\\tmp6j4gzjyl\\dummy_package\\pipelines\\my_pipeline\\nodes.py': O...\python.exe' -m build --sdist --outdir 'C:\\Users\\circleci\\AppData\\Local\\Temp\\tmpy65f9iu2\\dummy_project\\dist'\n" = <Result PermissionError(13, 'The process cannot access the file because it is being used by another process')>.output

Reverting the last patches and leaving this for now until I (or someone else) have more time to do some proper debugging.

[astrojuanlu]

In fact, what I'm going to do is to drop the micropkg changes and make this PR only about kedro package, as originally intended.