Unify docutils type annotations

[danieleades]

there are two sources of truth for docutils type annotations.

this repo uses docutils-stubs, but there's also typeshed annotations - types-docutils.

Neither are complete, nor is one a subset of the other. types-docutils was historically pretty poor but has seen more active development lately and has improved. doctuils-stubs is more complete, but is out of date and unmaintained.

I believe this repo should migrate to the typeshed annotations and upstream anything missing from docutils-stubs into typeshed.

cc @tk0miya

• improve type annotations in 'docutils.node.document' python/typeshed#11468
• improve type annotations in 'docutils.statemachine' python/typeshed#11469
• improve type annotations in 'docutils.transforms' python/typeshed#11471
• improve type annotations in 'docutils.parsers.rst.states.Inliner' python/typeshed#11473
• improve type annotations in 'docutils.parsers.rst.roles' python/typeshed#11481
• improve type annotations in 'docutils.readers' python/typeshed#11490
• improve type annotations in 'docutils.readers.doctree' python/typeshed#11492
• improve type safety when using NodeMatcher #12034
• improve type annotations in 'docutils.parsers.rst' python/typeshed#11523
• improve type annotations in 'docutils.utils.Reporter' python/typeshed#11521
• improve type annotations in 'docutils.node.document' python/typeshed#11468
• add 'paragraph' node to docutils python/typeshed#10102
• add docutils.nodes.General stub python/typeshed#10099
• fix annotation for directives.other.Include.run #12042
• improve type annotations in 'docutils.parsers.rst.directives.misc' python/typeshed#11524
• [WIP] improve type annotations in 'docutils.parsers.rst.states' python/typeshed#11525
• improve type annotations in 'docutils.utils' python/typeshed#11526
• improve type annotations in 'docutils.parsers.rst.tableparser' python/typeshed#11530
• improve type annotations in 'docutils.nodes.Element' python/typeshed#11539
• improve type annotations in 'docutils.io.Input' python/typeshed#11540
• improve type annotations in 'docutils.parsers.rst.states' python/typeshed#11545
• improve type annotations in 'docutils.parsers.rst.directives' python/typeshed#11597

and the actual PR to migrate to typeshed (WIP): #12012

[danieleades]

Have started tweaking type annotations in typeshed, with a view to reach parity and then transition this repo to typeshed

python/typeshed#11468

[picnixz]

Good idea! I am always frustrated because of how poor autocompletion is done with docutils on Pycharm so I'll be very happy if this can be merged upstream.

[danieleades]

i've created a PR to track the migration here - #12012

contributions (both to the PR and upstream in typeshed) very very welcome!

[danieleades]

@sphinx-doc/triagers @sphinx-doc/developers @sphinx-doc/former-maintainers @picnixz

can I pretty please get a solid review of python/typeshed#11469?

It introduces a generic type argument to the docutils state machine to represent the context. That generic argument will propogate through into a massive number of docutils types so is a significant change. Before that happens I want to check this is correct!
I'm not super familiar with the inner workings of the docutils statemachines. Is the 'context' truly generic? what is its type?

from the docutils docs:

context: application-specific storage.

[picnixz]

I won't be there next week, and I don't think I'll have time for that today. But I'll do it by in two weeks (I will take time for that).

[danieleades]

@sphinx-doc/triagers @sphinx-doc/developers @sphinx-doc/former-maintainers @picnixz

can I pretty please get a solid review of python/typeshed#11469?

It introduces a generic type argument to the docutils state machine to represent the context. That generic argument will propogate through into a massive number of docutils types so is a significant change. Before that happens I want to check this is correct! I'm not super familiar with the inner workings of the docutils statemachines. Is the 'context' truly generic? what is its type?

from the docutils docs:

context: application-specific storage.

stand down!

I've postponed adding generics, even if they are technically correct. The decision is blocking progress and would have significant downstream impact.
I'll revisit in a separate PR and just type the context as Any for now

[chrisjsewell]

Is the 'context' truly generic?

If you are talking about state transitions, like def blank(self, match, context, next_state):, then context is a list[str]

I'm not super familiar with the inner workings of the docutils statemachines

I am, I've been through it in excruciating detail, because I'm writing an RST parser as we speak 😉

This all looks great, thanks for all the work!
... although obviously it would be a million time easier, if we could just type docutils itself 😅

[danieleades]

If you are talking about state transitions, like def blank(self, match, context, next_state):, then context is a list[str]

I believe that's true of the rst-specific subclasses. Is it also true of the superclass? (I'll take your word for it)

[picnixz]

Ah thank you chris for helping here since I am travelling (I'm currently in a train). Could I ask you to help with the upstream PRs (I can come back later if there are some questions though it's been a while since I played with docutils)?

[chrisjsewell]

Could I ask you to help with the upstream PRs

Okie, I will try to carve out some time for it 👍

[danieleades]

i'm proposing that #12012 be merged, warts and all so that all future improvements to typing are made against the community-supported typeshed docutils stubs