Generating lvgl stubs using documentation

[kdschlosser]

Is your feature request related to a problem? Please describe.
Yes/No

Describe the solution you'd like
I was looking for a stub file for lv_micropython (MicroPython with LVGL) and I had come across this repo as it had a stub file that was generated. The stub file is incomplete, there are no type hints or docstrings included in the stub file. While it is nice to at least know functions and classes it is far from being a complete stub file.

I am not sure how the stub file is generated but I am guessing that a recursive iteration of the lvgl module is being done to generate the file. This would give the layout but not any of the types or documentation.

Describe alternatives you've considered
Using the generated stub file in combination with the doxygen XML output for LVGL to create a more complete stub file.

Using only the doxygen XML output I was able to generate this stub file.
https://github.com/kdschlosser/mp_lvgl_widgets/blob/main/lvgl.pyi

There are some pieces that are not done properly (enums nested into classes) that I am not able to get 100% correct using only the documentation. If your mechanism and what I have written were combined it would generate a complete stub file.

When compiling lv_micropython there is a json file that is generated that contains some information that can be used to generate a stub file, The issue is it is not complete and doesn't contain all of the information needed to generate the stub files properly. I have asked for it to be updated but I have not gotten very far in the motivation department for it getting done. The developer doesn't see the benefit of having a stub file generated.

The way I have gone about doing it was intended for my own personal use but since this repo existed I thought I would ask and see if there is any interest in combining what I have and what has been done here

[Josverl]

Most of the stub creation is fully automated so if you have a python script or something that can produce a py or pyi file then I should be able to integrate that in to the overall stubbing process

Or reading / visiting the Doc and parameter info could also work

[kdschlosser]

What I have is automatic and it does generate the stub file. Tho the stub file is not organized correctly. One of 2 things would need to be done. Either an iteration over the lvgl module in order to get the organizational layout or using the gen_mpy script and updating it so the JSON file that is output has the needed information for the organization of the module. The latter would be the better way to go about it. I can get any changes made to the gen_mpy script added to LVGL so there wouldn't be a need to keep any changes to the file in sync.

If you would be willing to make a branch on your I can add what I have and we can work together on getting it added. I don't know how you go about collecting the information and you don't know how my part runs so having something we are able to mess about with until we get it hammered out would be ideal.

That is if you are willing to work on it together. I have a lot of things going on so I don't have the time to learn how your code works but I am able to make changes and update the code that I wrote.

Just to let you know the code that I wrote isn't specific to LVGL. It can be used as a framework for collecting doxygen generated information and converting it into stub file format.

Another suggestion is to use the gen_mpy script that is in the LVGL binding to provide the structure/organization instead of iterating over the LVGL python objects.

I do have enough of a working knowledge of the script to update it so it provides the information that we need. The doxygen xml would still have to be accessed in order to do the type conversions, get values for enumerations and get the documentation. The hard parts are done, just a matter of tweaking them a little bit to get them to play nice with each other.

[Josverl]

Working together doubles the fun.

If you can share the script and result in a draft PR we can work through that.

Alternatively I can add you as a collaborator, and we can work on a fork.

I already have the functionality to merge info in two stub files, so perhaps that is all that's needed as post-processing

[kdschlosser]

I can have it dump the data in a flat manner like it is in C code only the enumerations would end up as classes.

[kdschlosser]

let me make the changes to the script so it outputs a stub of exactly how the C code is. The using your code to get the structure and assemble the stub file properly.

[Josverl]

indeed , if you can create a script that can generate one or more stub files for LVGL , enven if they are generic
then these can be merged with 'board_stubs' to to get accurate typings.

If needed the lvgl repo can be cloned in the repos\ folder to allow local access to the files. stubber clone --lvgl ?
Part of the challenge may be to support multiple versions of lvgl

also note a prior discussion on lvgl that had some thoughts on this.
Josverl/micropython-stubs#554 (comment)

%%{ init: {'theme': 'light','flowchart': { 'curve': 'bump' } }%%
graph TB

    %% Get frozen
    MPY --> |"fas:fa-folder-tree ports/.../boards/.../manifest.py"| GFS
    LIB -->GFS -->FR
    
    %% create firmware stubs
    MCU --> CS --> FW
    %% create docstubs
    DOCS --> GDS --> DS --> Merge
    %% create lvgl stubs
    LVGL/doxygen --> doxy_2_stub --> LVGL_STUBS --> Merge
    %% merge fw and docstubs 
    FW --> Merge --> MS

    %% Build 
    FR --> Build
    MS --> Build
    Stdlib --> Build
    Build <--> DB    
    Build --> PUB
    %% Publish 
    PUB --> Publish
    Publish --> PyPi --> PKG
    DB <--> Publish

    %% Styling 
    MPY{{fab:fa-github MicroPython}}:::repo
    LIB{{fab:fa-github MicroPython-lib}}:::repo
    DOCS{{fab:fa-github MicroPython Docs}}:::repo
    PyPi{{fas:fa-th-list PyPi}}:::pkg

    PKG[fa:fa-suitcase MicroPython-***-stubs]:::pkg

    PUB[/fa:fa-suitcase Packages\nfas:fa-folder-tree /publish/...//]:::stubs
    FR[/fas:fa-pencil Frozen stubs\nfas:fa-folder-tree stubs/... -frozen/]:::stubs 
    FW[/fas:fa-pencil Firmware-stubs\nfas:fa-folder-tree stubs/.../]:::stubs
    MS[/fa:fa-pencil Merged-stubs\nfas:fa-folder-tree stubs/... -merged/]:::stubs
    DS[/fa:fa-pencil Doc-stubs\nfas:fa-folder-tree stubs/... -docstub/]:::stubs
    Stdlib[/fa:fa-pencil Stdlib-stubs\nfas:fa-folder-tree stubs/-stdlib/]:::stubs
    
    DB[(fa:fa-database jsondb)]

    MCU[<img src='https://micropython.org/static/img/Mlogo_138wh.png'> MCU]:::device
    CS[[fa:fa-microchip createstubs.py]]:::stubber

    Merge[[Merge]]:::stubber
    GFS[[get-frozenstubs]]:::stubber
    GDS[[get-docstubs]]:::stubber
    Build[[Build]]:::stubber
    Publish[[Publish]]:::stubber

    classDef repo fill:lightblue,stroke-width:4px
    classDef stubber fill:lightgreen
    classDef stubs fill:cyan,stroke-width:4px
    classDef pkg fill:cyan,stroke-width:4px
    classDef device fill:black,stroke-width:4px,color:white,stroke:white

%% update

[kdschlosser]

the versions don't matter at all because the script I wrote doesn't care about the version. It reads the XML files generated by doxygen when building the documentation for LVGL. It does this in a manner that is not version specific. I would think that doing a recursive iteration over the lvgl module in micropython would not have anything version specific to it either.

Something along these lines is what I have used in the past to iterate over the lvgl module in MicroPython to generate a basic stub file.

import lvgl as lv

def iter_lvgl(obj, indent=''):
    for item in sorted(list(dir(obj))):
        if item.startswith('__'):
            continue

        value = getattr(obj, item)

        type_ = type(value)

        if 'function' in str(type_) or 'bound_method' in str(type_):
            if indent:
                print(indent + 'def ' + item + '(self, *args, **kwargs):')
                print(indent + '    ...\n')
            else:
                print('def ' + item + '(self, *args, **kwargs):')
                print('    ...\n')

        elif 'type' in str(type_):
            print(indent + 'class ' + item + ':')
            iter_lvgl(value, indent + '    ')
            print()
            if not indent:
                print()

        elif 'class' in str(type_) and 'int' in str(type_):
            print(indent + item + ': int = ...')

        elif 'NoneType' in str(type_):
            print(indent + item + ': Any = ...')


        elif 'lv_obj_class_t' in str(type_):
            print(indent + 'class ' + item + ':')
            iter_lvgl(value, indent + '    ')
            print()
            if not indent:
                print()

        elif 'Blob' in str(type_):
            print(indent + item + ': lv_obj_class_t = ...')


        else:
            print(item, type(value))

The above is not complete but an example. Not version specific and gives the basic structure. It wouldn't be that hard to reverse the naming conventions used in the binding so a match could be made with the output from my script.

[kdschlosser]

idk If you looked at this or not.

https://github.com/kdschlosser/mp_lvgl_widgets/blob/main/lvgl.pyi

This is what the script I wrote generates. almost exactly what you see there. I did make a few manual changes but nothing substantial. If you can use that to build a complete and proper stub file of LVGL then we are good. I am not 100% sure if all of the functions are where they belong a manual check would need to be done to see what is out of place. It might be easier to flatten it out so everything is at the module level and then use what you have to locate the parts and put them where they belong instead of having to locate things I might have in the wrong locations.