missing feature: multithreading libs (like Due Scheduler)

[shiftleftplusone]

hello,
as these 2 Arduino multithreading libs
https://github.com/arduino-libraries/Scheduler (Zero, M0, Due)
https://github.com/mikaelpatel/Arduino-Scheduler (professed: AVR+ARM core)
do not work with ESP8266, which multithreading libs do you provide which work for the ESP8266 nodeMCU?

[lrmoreno007]

Ticker.h are directly inside this project, more info https://arduino-esp8266.readthedocs.io/en/latest/libraries.html#ticker

Ask these types of questions at www.esp8266.com

Please close this. It isn't a issue.

[shiftleftplusone]

no, I ask this question here because it's about a ported lib especially for the esp8266, such as the special libs about SD, SPI, Servo, Wire.
The missing library is the issue!
After all, @lrmoreno007 , I do not see either working multithreading lib in your links though, so what is the solution to my topic?

[shiftleftplusone]

so no multithreading available for the nodeMCU like the (cooperative) Scheduler or even a pre-emptive MT system, to be used as a lib with the standard Arduino IDE?

[devyte]

@shiftleftplusone the problem to implement this is dealing with the wifi stack context, which you don't need to do on Arduino or other devices. In the ESP, the wifi context is inside the espressif SDK.
I suspect it may be possible to do with some massive rewrite and reverse engineering of many low level routines, but that's just a guess, and the very few people who have the knowhow probably don't have the time or need.
Care to tackle it yourself?
Looking at the source code for the ESP's yield() and delay(), as well as the implementation of the libs you mention would be a starting point.
My engineering spidy sense hints at treating the wifi context as special, and treating all the loop contexts the same. When a loop context yield()s or delay()s, you store that loop's context and go back to the scheduler, who then immediately services the wifi. Then, once done, it chooses a loop of those available to continue. Rinse and repeat. Keep the contexts together with the loop info inside the scheduler.
You'd have to think about maintaining compatibility when the scheduler lib is not used, to avoid mem overhead in that case.

Async code executed from callbacks would still be forbidden from calling yield and delay.

The above is off the top of my head, and without deep knowledge of the code specifics, but I think it's doable.

[shiftleftplusone]

hi,
I am just an Arduino end user and I have actually absolutely no skills in C++ programming, WiFi contexts or whatever, or how MT for common Arduino is done (e.g., by some existing Schedulerlibs e.g. by cmaglie or mpatil for the Arduino Due). Nevertheless, multithreading is absolutely indispensible: WiFi can run in the background, and multiple threads (e.g. sensor watching by different polling speed via i2c, UART, SPI and controlling actors like motors and LEDs) must be able to be run simultaneously in parallel threads.

[devyte]

There are ways around what you seem to need without the Scheduler library. I am knowledgable in the field of automatic control, and I can tell you that you can implement control loops of differing frequencies, without multithreading. You just need polling on the cpu ticks and a roundrobin checker in the loop().
As an alternative, you may want to look into the FreeRTOS version of the ESP espressif SDK for real multithreading.

But let's not hijack this issue. You requested an implementation for the Scheduler. I would love to see it happen, but there doesn't seem to be anybody willing to pick up the request, so that leaves just you.
I could show you where to start looking. The code doesn't seem exactly easy to understand, but it's not big and broken into small pieces, which is always good. I also see a context container, currently of size 1, which would probably just need to be replaced, or extended and managed. There's also some docs from espressif.

If you're still not willing/able to pick up your own request, and learn interesting things about the ESP in the process, then please close the issue.

[shiftleftplusone]

I have been using the Scheduler libs by cmaglie for my Due and also the POSIX pthread libs for my Raspberry Pi (ANSI C, not C++) already very successfully since a lot of years and before that the MT libs of some different bytecode intepreters since almost 20 years for Lego Robots - I know how to handle multithreading, truely. But I never wrote MT libs from the scratch, neither for the Due nor POSIX libs for the Pi nor for Lego bytecode interpreters ever, and will honestly never do that for neither platform, and finally not for the nodeMCU: As stated, I am an end user, I expect those libs to be available for either platforms, ready-to-use for customers like me. So if no MT for the ESP8266 will be available, I will have to wait for them to come - or drop that node MCU.

[suculent]

The purpose of microcontrollers is to drive circuits on time. Multitasking and async processes do not have expectable timing and therefore do not fit such purpose and cannot be used in machine-control. You should select different core like RPi which has much more processing power. Moreover ESP8266 is a development platform and is not directly intended to be part of an end-user UX. Odesláno z iPhonu 4. 9. 2017 v 9:46, shiftleftplusone <[email protected]>:

…

I have been using the Scheduler libs by cmaglie for my Due and also the POSIX pthread libs for my Raspberry Pi (ANSI C, not C++) already very successfully since a lot of years and before that the MT libs of some different bytecode intepreters since almost 20 years for Lego Robots - I know how to handle multithreading, truely. But I never wrote Scheduler libs for the Due or POSIX libs for the Pi or bytecode interpreters for Lego bricks ever, and will honestly never do that for neither platform, and finally not for the nodeMCU: As stated, I am an end user, I expect those libs to be available for either platforms, ready-to-use for customers like me. So if no MT for the ESP8266 will be available, I will have to wait for them to come - or drop that node MCU. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[shiftleftplusone]

you seem not to be very experienced with the Arduino Due (ARM Cortex M3) - the (cooperative) MT Scheduler works fine and is indispensible for Robotics. Actually already the availability of different Scheduler libs (Patil, Maglie, Jensh, Pessaux, and more) both for the Due, M0, and partially even for AVR cores is proving it's eligibilily and legitimacy. The nodeMCU is almost as powerful as the Due , and much more powerful than AVRs and even better than M0 SAMD cores, and so MT libs are expected to be available, too, and thus to fit into the Arduino IDE concept plan for the Arduino target group.

[suculent]

You’re right. I’ll look at those first. I’m using mostly Extensa LX6 cores, most Arduinos are to slow for me and lack WiFi.

…

On 4 Sep 2017, at 10:01, shiftleftplusone ***@***.***> wrote: you seem not to be very experienced with the Arduino Due (ARM Cortex M3) - the (cooperative) MT Scheduler works fine and is indispensible for Robotics. Actually already the availability of different Scheduler libs (Patil, Maglie, and more) both for the Due, M0, and for AVR core is proving it's eligibilily and legitimacy. The nodeMCE is almost as powerful like the Due , and much more powerful than AVRs and even better than M0 SAMD cores, and so MT are expected to be available, too. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3390 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANix0wVZN77XUgpIo_d55Ip4yBgZkFUks5se65PgaJpZM4OLRyu>.

[suculent]

https://github.com/anmaped/esp8266-scheduler https://github.com/nrwiersma/ESP8266Scheduler Have you tried any of those? Odesláno z iPhonu 4. 9. 2017 v 10:01, shiftleftplusone <[email protected]>:

…

you seem not to be very experienced with the Arduino Due (ARM Cortex M3) - the (cooperative) MT Scheduler works fine and is indispensible for Robotics. Actually already the availability of different Scheduler libs (Patil, Maglie, and more) both for the Due, M0, and for AVR core is proving it's eligibilily and legitimacy. The nodeMCE is almost as powerful like the Due , and much more powerful than AVRs and even better than M0 SAMD cores, and so MT are expected to be available, too. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[devyte]

@suculent thank you for those links, I was not aware of those libs.

The first looks unmaintained for a couple of years, but looks rather advanced.
The second looks to be in active development. However, I find the fact that execution never reaches the main loop to be very scary.
I'm not sure how the interface compares to the standard Arduino Scheduler. Do we care about compatibility?

@shiftleftplusone care to try them both out and report back?

[suculent]

Examples and Arduino core system parts should be working on all models, which is kind of an argument against having this inside core OS library. However, @shiftleftplusone mentioned the major difference in ESP8266 and ‘standard’ Arduinos. In future, I’ll definitely use anything faster like ESP8266 (I have 20 of those), ESP32 or ILI7697. Arduino platform should help educating on multi-threading rather than being stuck to 8MHz Atmels (I have one in Arduino format and several nanos that are pretty useful, but WiFi-less).

…

On 5 Sep 2017, at 06:03, Develo ***@***.***> wrote: @suculent <https://github.com/suculent> thank you for those links, I was not aware of those libs. The first looks unmaintained for a couple of years, but looks rather advanced. The second looks to be in active development. However, I find the fact that execution never reaches the main loop to be very scary. I'm not sure how the interface compares to the standard Arduino Scheduler. Do we care about compatibility? @shiftleftplusone <https://github.com/shiftleftplusone> care to try them both out and report back? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3390 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANix0whMiw6uUZ0i1nZzaKiVn3j3DKBks5sfMgEgaJpZM4OLRyu>.

[shiftleftplusone]

hello,
the anmaped/esp8266-scheduler seems to be reasonable as it's forked from mikaelpatel/Arduino-Scheduler which I once already tested on my Due. Although some issues have already been reported for the original fork run by Dues (opposite to the cmaglie scheduler which always works fine) it seems to be the more promising approach for the esp8266 IMO.

The 2nd one (https://github.com/nrwiersma/ESP8266Scheduler) using classes is too weird tbh.

Most promising for the future would be an approach based on POSIX pthread ported to Arduino though.

I'll have a look at the mpatil(anmaped scheduler and report ASAP.

[shiftleftplusone]

edit:
just observed that this anmaped/esp8266-scheduler has the same lib name "Scheduler" ( #include <Scheduler.h> ) as the cmaglie and the mpatil version which is already installed on my system and would overwrite them immediately - so it's no choice unless there would be a way to rename it before (e.g., #include <esp8266Scheduler.h> )

[shiftleftplusone]

the anmaped Scheduler
https://github.com/anmaped/esp8266-scheduler
does not compile for the esp8266 at all (tested e.g. SchedulerBlink.ino) even when all different Scheduler libs have been removed from the libraries folder before.

[5chufti]

the esp8266-scheduler you mention needs heavy patching of the Arduino esp8266 core based on a very specific level (4897e00).
I think this circumstance is missing in the "Installation instructions" and might explain your problems.

[anmaped]

@shiftleftplusone It works. You should use the same Arduino branch, as I reported before. I don't have time to update it to the last esp8266 Arduino. If someone could help me I can do it. I saw that several people are interesting.

[shiftleftplusone]

I don't speak C++, just simple C99 syntax.
As read somewhere else, it is supposed to be to be restricted to IDE 1.6.8, but 1.6.8 is outdated, and for current esp8266 projects including different other libs I also always use 1.8.5
So the issue is still unresolved for current Arduino IDEs.

[anmaped]

@shiftleftplusone You have the instructions here (https://github.com/anmaped/esp8266-scheduler). We only support 1.6.8.

[d-a-v]

@shiftleftplusone

@dok-net would like to notify you because you might be interested.

In January 2018, you were asking about the Arduino Scheduler lib. The cores/esp8266/Schedule.h existed back then, I wonder why you didn't notice and nobody pointed it out for you. It's not the same exact API, but can do many of the same things for you, I guess.

Thank you @dok-net for this heads-up (ref)

[dok-net]

So here's something that I have cooked up that works right now on ESP8266 Arduino master, at least for me. No guarantees, just a pet project.

You write tasks as loop functions - for (;;) or while (true) at top level - and place you sequential code in there, each task attending to a single (repetitive) concern. To give CPU time to other tasks or delay for specific amounts of time, there are yield() and delay(). A task can also be terminated by exit().
In the standard loop, you place a run() call for each of the CoopTask objects.
The hard part is in figuring out how much stack each task absolutely needs in order to not crash the MCU.

The included examples blinks the builtin LED, responds to interrupts from a button press, and runs a webserver.

What is different to the Schedule API in ESP8266 core? IIRC, yields don't switch between the scheduled functions, they just keep the watchdog timer and other internals happy. So for interesting program activity, you have to write a lot of separate small functions and try to schedule them in the right order. That said, you should be able to turn responsibility for the run() functions to Schedule, but YMMV.

Enjoy: https://github.com/dok-net/CoopTask.git

[earlephilhower]

We now have scheduled functions and the Ticker to give some of this functionality, for those looking at this bug anew.

[dok-net]

@shiftleftplusone I, too, came across the Arduino Scheduler library while researching cooperative multithreading/tasking for the ESP8266/ESP32. Writing sequential sketches that explicitly yield() between them is a) much simpler than preemptive threading, b) a lot like real-time OS tasks that usually yield between each other, at the same priority, except that there is a preemption compile time option in FreeRTOS, whatever that exactly does.
Given this situation, I've implemented the CoopTasks library that I mentioned earlier, it's at release 1.0.0 now, get it from CoopTask GitHub repo or the Arduino library manager.
As it runs across ESP8266, ESP32, Arduino (tested: Pro Mini) and Windows (currently only x86 32bits) and Linux, the API allows writing quite portable tasks.

@earlephilhower I really need to point out that the Arduino Scheduler library and the CoopTasks library are not quite the same as the ESP8266 scheduled functions and the Ticker. Both of the latter allow only for stackless coroutines. In fact, CoopTasks uses the ESP8266 scheduled functions to implement stackful coroutines, a.k.a. loops or tasks. Please really have a look - and if you've time available, please review #6307.

[dok-net]

@devyte Coming from the author of CoopTask, having built this library for the same purpose summarized in this issue's subject line, this may be subjective, but I'd suggest this issue can be closed for good.
I recommend reading the Wiki entry about memory use.