Testing trasmitter and receiver

[CaliCaliIT]

Using flight random sketch trasmitter on ESP32, seems that only BlueTooth trasmission is decoded correctly as OpenId protocol bye receivers (Tested on Android Receiver and Id_scanner).
WIFI protocol instead is trasmitted on channel 6 but not showed as OpenID from the receivers. I Captured the Wifi Signal and also the wireshark dissector is not able to decode it correctly as OpenId protocol.

Did someone test the ESP32 trasmitter on WIFI (beacon o nan)?
Any hints?

Thanks

[sxjack]

Unfortunately I do not have an Android device that will receive opendroneid WiFi beacon or NaN. My ESP32 scanner picks up both beacon and NaN from the library. I checked this yesterday when I updated everything for opendroneid 1.0.

What are you using as a WiFi receiver?

I will have a go with Wireshark tomorrow, but it's on an ancient netbook and can be a bit temperamental.

[CaliCaliIT]

Im tried both an android receiver and the ESP32 scanner of your project. In particular for the last case i used one ESP32 to trasmit with random_flight code and one to receive with id_scanner. All ok with Bluetooth but not with Wifi. Same behaviour if i use the android App as receiver.
Did you try already to send and receive with the ESP32 ?
I'll keep to analyse with wireshark to understand what could be wrong.

[sxjack]

I just ran the random flight example on an ESP32 transmitting both BLE and WiFi beacon and my Moto G7 picked up the BLE and my scanner (the one in the photo on this repository's home page) picked up the WiFi.

Have you tried it with the library just transmitting WiFi?

[friissoren]

What version of the android receiver do you have installed? Make sure you have the latest.

Any phone with Android 6 or newer can pick up the Wi-Fi Beacon signals.

Reception of Beacon signals is significantly slower than Bluetooth. Patience is required. If you haven't disabled Wi-Fi throttling it can take 2+ minutes to pick up a ping. Sometimes much more.

Did you enable the compile flag for Beacon in the ESP32 code?

[CaliCaliIT]

Yes i tried to enable just the WIFI define and i see the Wifi beacon tramission on channel 6 coming from the ESP but is not showed up neither from the ESP or the android. I tested successfully what you have depicted in your photo but using the BT4, Showinf the info on the Drone in the Android app it shows is using BT to decode (see photo below). if I disable the BT on the trasmitter, the drone won't be detected anymore.

[sxjack]

Any phone with Android 6 or newer can pick up the Wi-Fi Beacon signals.

My Moto G7 which is Android 10 doesn't see the beacon frames. The app (3.0.0) says that beacon scan is supported.

[CaliCaliIT]

On my samsung (Android 11) app (3.0.0) says wifi beacon and nan are supported. Still cant receive the wifi beacon from esp. Il Mer 20 Ott 2021, 20:59 Steve Jack ***@***.***> ha scritto:

…

Any phone with Android 6 or newer can pick up the Wi-Fi Beacon signals. My Moto G7 which is Android 10 doesn't see the beacon frames. The app (3.0.0) says that beacon scan is supported. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AILYTFMST6ECSJB6MVVJLBDUH4GPRANCNFSM5GLGJSOQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[friissoren]

Hmmm.....

Beacon reception should be supported for both of your phones. What happens if you disable Wi-Fi throttling?

Do you have a Linux PC? Could you try the Beacon transmitter based on hostapd:
https://github.com/opendroneid/transmitter-linux

[sxjack]

Beacon reception should be supported for both of your phones. What happens if you disable Wi-Fi throttling?

Doesn't work with throttling on. Does work (although it is not as good as BLE) with throttling off. (I thought that I had throttling off, but it was on...)

[friissoren]

I have seen something similar with the OnePlus 6T. With throttling it would sometimes take 10+ minutes to get a single ping through. Without throttling the reception rate is somehow reasonable.

[sxjack]

With throttling it would sometimes take 10+ minutes to get a single ping through.

I probably left the G7 for more than 10 minutes throttled without seeing anything. Unthrottled, it varies between a few seconds and a minute between frames.

My Android 6 tablet says that it supports beacon, but I haven't see it pick anything up. I can't see an unthrottle option in its developer options. I'm not really surprised by this, it only just works with BT4.

I had Wireshark running this afternoon on my trusty old NC10. I could see the beacon frames, but I can't get the dissector to work.

CaliCaliIT

Which ESP32 device are you using? I have the basic WROOM dev board and a TTGO mini 32.

[CaliCaliIT]

I'm using WROOM ESP32. Same model to run the transmitter and the receiver.
I'm writing a new wireshark dissector and in the weekend i'll do other test and i'll let you know the results....

[CaliCaliIT]

The dissector on GitHub is not working on the ESP beacon captured with Wireshark that adds a RadioTap header. That's the reason im working on the dissector.

[CaliCaliIT]

Here it comes the result of analysis of why Dissectors and ID Scanner did not recognize the WIFI Beacon coming from the ESP32 trasnsmitter.
Wireshark dissector in GitHub is rigid in the headers size and expected a RadioTap header size different from what is captured from ESP.
I created 2 new dissectors (one for WIFI and another for BT) that do not rely on the size of the Headers. Both now are able to recognize correctly the WiFi Beacon and BT Legacy protocol trasmitted from ESP32.
After that i worked on the problem on the IDScanner receiver and i discovered it was not listening on channel 6 but 10.
Changing esp_wifi_set_channel(6,(wifi_second_chan_t) 10); in esp_wifi_set_channel(6,WIFI_SECOND_CHAN_NONE); now I finally have an ESP transmitting and the other receving WIFI Beacon correctly.

After that i tried to test Wifi-NAN tramission with ESP32 and analyzing the WIFI signals on Channel6 I found only SYNC BEACONS but no SDF ACTION FRAMES coming from the ESP32. Any idea or hints on that? Did you test that also?

[friissoren]

I have not tried to use the ESP32 as a receiver, neither have I ever managed to get the Wireshark capture working (I can't get monitor mode to work on any of the HW/SW platforms I have tried).

However, the NAN signal that the ESP32 transmitter sends is possible to be captured by the Android Receiver application on a Samsung Galaxy S10 phone.

[sxjack]

After that i worked on the problem on the IDScanner receiver and i discovered it was not listening on channel 6 but 10. Changing esp_wifi_set_channel(6,(wifi_second_chan_t) 10); in esp_wifi_set_channel(6,WIFI_SECOND_CHAN_NONE)

That is interesting because the ESP32's that I have seem to ignore the second channel setting.

After that i tried to test Wifi-NAN tramission with ESP32 and analyzing the WIFI signals on Channel6 I found only SYNC BEACONS but no SDF ACTION FRAMES coming from the ESP32. Any idea or hints on that?

I get the NAN frame from the opendroneid code so you would need to ask over there.

[sxjack]

A correction to my earlier comment.

You should see two frames, sync followed by action. The content of the frames comes from the opendroneid code.

[CaliCaliIT]

That's exactly what i was expecting to see. One sync beacon frame followed by an SDF action frame.
On the sniffer side i see only the Beacon one sent by the ESP. do you receive both in your tests? I'll keep try to investigate with other tools.....

@friissoren on my S20 (declared from the app itself as compatible for all the tx/rx mode) i see only BT signals. KO both WIFI beacon and NAN from ESP32. Are you able to see them on your S10?

[friissoren]

On the Galaxy S10, I can receive all three signals: BT4, NaN and Beacon. As far as I can see from the git history or my local copy, the code I have running on the ESP32 is from the end of May this year but I guess there shouldn't have been anything after that which would have broken things.

The reception of the Wi-Fi signals is clearly slower than Bluetooth so try to wait a few minutes.

Try also to disable the Wi-Fi throttling on your phone to see if that helps:
https://github.com/opendroneid/receiver-android#wifi-beacon

[friissoren]

Were you able to get this working?

I discussed with one other person who has a Samsung S20 FE 5G phone and he has been able to pick up Wi-Fi Beacon signals from one drone manufacturer's transmitter implementation (not ESP32) using app version 2.3.0 (I don't think there should be any difference related to this in the app compared to v 3.0.0).

Recently, there has been a function added to the core-c code for generating the Wi-Fi Beacon frame data. It is not integrated with the sender sample program but you could try to compare the code against the Wi-Fi Beacon code in the ESP32 repository?

[CaliCaliIT]

@friissoren , i was focused on building a Linux receiver that I tested on the ESP32 trasmitter.

Did the person you talked with has recompiled the source or did install an available APK?
If he/she had compiled the source, what target android sdk did use? I compiled the 3.0 APK for android 11.
I will restart debugging it in the weekend.

Do you know which drone manufactur has DRI implementation that can be tested? I would like to test the receiver.....

[friissoren]

I don't know who exactly compiled the Android app for him but that shouldn't really matter since all the various part of the Android SDK that are needed are specified by exact version in the source code of the application.

I am not aware of any publicly available drone ID implementations from any drone manufacturer. I think everybody is still in wait mode until the rules and standardization settle down and it becomes mandatory to start broadcasting these signals. That is one of the main reasons for sharing the various opendroneid code repositories on Github in order for people to have something to start from.

Did you try to disable the Wi-Fi throttling in the phone?

Do you have access to a PC running Ubuntu? You could try to the hostapd based transmitter, just to sanity check that the phone is okay.

[sxjack]

Going by Tissy's comment in issue #7, Parrot Anafis are transmitting IDs.

I got the sender program to work. I can't remember if it was on my NC10 or Raspberry Pi (or both).

If you are going to be talking about the Android app, it might be worth raising an issue over in the app's repository.

[CaliCaliIT]

Ciao
I finally got some results on testing the Wifi Beacon ESP32 trasmission trying to understand why is not recognized by the Android app. Here, for you utility, is what I found. There are 2 different problems in the trasmitter related to the way the APP receives the OpenID signals. Debugging the code I noted that the app seems to scan the list of the standard APs around. It has two impacts on the transmitter:

1. the trasmitter needs to present itself as visible AP and not hidden as AS-IS in the code.
2. the trasmitter needs to send a beacon frame with all the fields that are mandatory in an Access Point. Current code is missing inserting some of those fields.

My tests reveals that if the 2 points above are not satisfied, the app ignore the frame received, while if they are the drone is shown with the correct data !
There is still something wrong cause i noted that the visualized drone does not move and seems not to be updated in position (or at least is updated once in while very rarely). Should be related to the frequency of the wifi scan of the OS and not to the ESP32 trasmitter.
It needs to be further investigated, but at least now the WIFI Beacon of the ESP32 transmitter is recognized by the app.

@friissoren , yes I tested with success the WIFI Beacon linux transmitter with the app. Since the transmitter is using hostAPD to send the Beacon Frame, it naturally satisfies the req 1) and 2) above. Unfortunatelly I could not test the linux trasmitter on the update problem i saw with esp32, because it sends a static drone position.

Sorry for the long message :)

[friissoren]

Thanks for updating. Good to hear that you have figured out what was wrong. There would probably be some benefit if you are able to share the changes you did to get things working? I assume that @sxjack would not mind receiving Pull Requests?

In the app, you can follow the update rate on the Info screen, by looking at the message delta field, showing the time delta between the last two received messages. From the hostapd transmitter it would typically be 2.4s (or some multiple of that when it misses a message).

Your findings would explain the difference I have seen in the receive rate on the Samsung Galaxy S10 and the OnePlus 6T. The S10 picks up the Wi-Fi Beacon from the ESP32 almost immediately but the 6T has to wait minutes between picking up a message squirt. Apparently the Wi-Fi driver in different phones act differently to hidden APs.

BTW: About the fields you refer to in 2), are those all present in the new function for Wi-Fi Beacon that has been added recently to core-c by Sensefly? I don't believe that function is yet used anywhere but it might be useful to help fix the issues you are seeing?

Sorry for keep asking but you never confirmed whether you are receiving with or without Wi-Fi throttling disabled in the phone.

[friissoren]

I took the latest code that Steve has updated to use the odid_wifi_build_message_pack_beacon_frame() function and I commented out the line wifi_config.ap.ssid_hidden = 1; in id_open.cpp.

Unfortunately, I still see the same behavior as earlier. On the Samsung Galaxy S10, reception of the beacon frames is somewhat reasonable (2.6s intervals at best, typically somewhere between 10 - 20 seconds) but on the OnePlus 6T reception is very rare (sometimes less than a minute between pings, sometimes significantly more). This is when I have the transmitter and the phones next to each-other.

@CaliCaliIT: It would be interesting to hear what it is you found related to your item 2) above:

the transmitter needs to send a beacon frame with all the fields that are mandatory in an Access Point. Current code is missing inserting some of those fields.

What exact fields are missing?

[pierreSCpapon]

I have been following this issue for a while and done some tests as I also face the same issues.
I can confirm the frequency of update is in the minutes with a pixel 4 and 15-30min on other older devices.
This is with the latest changes made by @sxjack .

I am also interested in the changes made by @CaliCaliIT in order to do some testing on my end.

I have made some wireshark captures of a Parrot drone known to broadcast RID. If it is useful, I can post content of the capture for comparison. The Parrot reports well on the app.

Note "Last seen" time

[sxjack]

I have made some wireshark captures of a Parrot drone known to broadcast RID. If it is useful, I can post content of the capture for comparison.

That would be useful. If you do so I will try to duplicate the Parrot frame. I only need the bit up to fa 0b bc.

[CaliCaliIT]

unfortunately I did not have the time to test more the ESP32 stuff :(
in any case to answer @friissoren and @pierreSCpapon, the mandatory field in an AP beacon are:

Timestamp
Beacon Interval
Capability info
SSID
Supported Rates

in the code, the beacon header was constructed "manually" and it was missing the supported rate fields.
I added that, I disable the hidden feature of AP (wifi_config.ap.ssid_hidden = 0) and magically my S20 started to receive both BT and WIFI beacon from the ESP32. Clearly shown as 2 different drones :)
Too bad the rate of refresh of the WIFI BEACON is quite toooooo slow, but I guess it depends from the refresh rate of the AP list on my S20,

regarding the AP mandatory fields of the beacon, probably the best way is using the the new constructor for Beacon msgs in the core library cause all the fields are present.

@pierreSCpapon I tested the parrot drone of a friend of mine but it does not have and is not transmitting. any of the openId protocols of the specs. What is the model you use? was trasmitting wifi or BT? Should be useful to have the cap capture so I can test my dissectors,......

P.S: I finally received Ev Board for BT 5.1. LR SOC and I'm going to start soon (if I'll have some spare time) to test also that protocol in OpenID. keep you updated,

[sxjack]

the mandatory field in an AP beacon are:

Timestamp Beacon Interval Capability info SSID Supported Rates

Supported Rates is not mandatory according to the reference that I have.

Anyway, I added it and it doesn't seem to make a lot of difference. The time between my phone seeing ODID beacon frames can be anything from milliseconds to a minute.

[pierreSCpapon]

Here are captures of a Parrot ANAFI broadcasting RID. I included captures from 2 different WiFi cards in monitor mode as I found the ALFA card wasnt picking up that many packets.
I also included a screenshot showing the ESP32's packet next to the Parrot's.

Let me know if this is sufficient as I had some trouble getting monitor mode to work so I want to be sure I captured what you need.

FYI the MAC address of the Parrot is 90:3a:e6:5b:dc:74 using wlan.ta == 90:3a:e6:5b:dc:74 as filter in wireshark might be helpful.

I would be interested in testing out your dissector @CaliCaliIT as I cannot get opendroneid/wireshark-dissector to work. (perhaps my monitor mode issue), maybe you can take my captures and try?

Lastly, in order to get RID to work with the Parrot ANAFI, I had to turn on the option in Preferences>Network>Broadcast DRI: ON. I do not have a valid operator reg. number but leaving it blank still worked.

If my captures are NG, let me know. I can work on the issue and do another round.

cap.zip

[sxjack]

@pierreSCpapon Neither of the frames in your screen shot are particularly interesting, but I did find one that is.

In this frame Parrot are saying channel 4 and in the one that you screen shot they are saying 11.

[gabrielcox]

The dissector on GitHub is not working on the ESP beacon captured with Wireshark that adds a RadioTap header. That's the reason im working on the dissector.

I have an update I will provide to accommodate differing header sizes. I'll send this tonight.

If anyone has an issue with the dissector? Please feel free to create an issue in the dissector repo (along with a .cap file).

[CaliCaliIT]

@pierreSCpapon i analysed your captures with the OpenID dissectors and the WIFI Beacon protocol messages were found correctly. You captured a lot of wifi traffic so I had to use filter to select just broadcasts from the parrot. I was expecting to find them just in channel 6 but the situation is different. I need to check better the specs.
In any case here below an example screenshot for both of your two cap files.

I noticed in the data you used a Parrot ANAFI. Do you know if the feature is available in all the versions of ANAFI?

@gabrielcox good to know you are working on the dissector to fix the header size problem. Let me know if you want to compare the updates.

[CaliCaliIT]

@sxjack

Supported Rates is not mandatory according to the reference that I have.

I don't know what specs you are using but in the 802.11 specs I have that "Supported rates" Is one of the mandatory field in the frame body of a beacon.
Here below the diagram:

In any case adding it did resolve the no-show on my mobile but, as I said in my old post, is not related to the Android receiver refresh rate problem that seems everyone is experiencing with the esp32 transmitter.
Should be interesting to know from @pierreSCpapon if the had noticed the same Android very slow data refresh with the WIFI beacon protocol transmitted by the Parrot.

[friissoren]

I have never been able to get the Wireshark capture working properly (i.e. get it into monitor mode on any of the platforms that I have tried). However, the two captures that Pierre uploaded are very interesting and I found out that using the filter wlan.tag.oui == 0xfa0bbc (i.e. the ASD-STAN OUI) you can get just the messages that contain drone ID data.

You can see the raw drone ID byte data by opening IEEE 802.11 Wireless Management -> Tagged Parameters -> Tag: Vendor Specific: Asd-stan.

There are a lot of other (I guess optional?) elements in those same beacon transmission squirts. If we can figure out what it is that is missing in the ESP32 transmitter beacons compared to the Parrot/hostapd transmissions, it might be possible to get the ESP32 transmissions to be more easily picked up by the receiver phones?

It is very clear, that when I use the hostapd transmitter from my Ubuntu PC to transmit, the Samsung Galaxy S10 picks up a ping almost every ~2.4 seconds and the OnePlus 6T almost every ~7.2 seconds (both phones have Wi-Fi throttling disabled). But when I use the ESP32 transmitter (with the odid_wifi_build_message_pack_beacon_frame() function in use and commented out the line wifi_config.ap.ssid_hidden = 1; in id_open.cpp.), the S10 will pick up a ping anywhere between ~8 - ~40 seconds, whereas the 6T will almost never pick anything up (there can be many minutes between). The combination of ESP32 + OnePlus 6T is almost on the borderline of being useless.

@pierreSCpapon: Is there any possibility of you also being able to capture transmissions from a hostapd implementation and adding the capture to this discussion?

If we can figure out what the delta is, even if it is just a matter of the transmission interval being different in the two implementations, we can cross-check that against the drone ID standards. If something is missing in the standards, @gabrielcox and I can try to get that corrected still, before they get published. I am very glad that @CaliCaliIT started investigating this and that you are all contributing to try to solve this mystery. This clearly shows the power of open sourcing what we have available.

FYI: I documented here what the standards say about the transmission interval. Beyond that, the standards doesn't really specify anything else than you must use the IE211 Vendor Specific tag with the ADS-STAN OUI value and the Vendor Type of 0x0D. It is possible that we would need to add some additional details to ensure all receivers can pick up the transmissions at a reasonable rate.

[friissoren]

I still can't get Wireshark to capture anything but I did manage to use the "Acrylic Wi-Fi Professional" app in trial mode to capture a few raw packets from both the hostapd and the ESP32 transmitter. There didn't seem to be any way to dump that data in an easily comparable format but I got at least some screenshots created:
ESP32_vs_hostapd.zip

Looking through those, there are some differences in the elements that they are sending. Maybe some of you are able to extract some useful data from this? I will try to take a deeper look as well.

I did manage to dump the data to a pcap file as well, which Wireshark mostly opened. However, something is different between the two transmitters because Wireshark recognize the drone ID element from the hostapd transmitter but not from the ESP32 transmitter. It is complaining about some elements that it doesn't yet know how to decode.
Beacon_capture.zip

[friissoren]

Hmm.... Update. I looked in more details at the ESP32 captured data and those beacon frames doesn't even contain drone ID data (i.e. no ASD-STAN vendor specific element). It seems that my laptop has exactly the same problem as the phones, it is not able to pick up the correct beacon frames.

What is strange is that it does regularly pick up some beacon frame from the ESP32. Either it doesn't know how to fully decode that data or the ESP32 sends out two types of beacon frames (with different interval?) and only the "slow" one of those has the drone ID data in it?

[sxjack]

What is strange is that it does regularly pick up some beacon frame from the ESP32. Either it doesn't know how to fully decode that data or the ESP32 sends out two types of beacon frames (with different interval?) and only the "slow" one of those has the drone ID data in it?

There are two sets of beacons. The ones from the underlying firmware because it is configured as an access point and the ones with ODID data in.

I had another look at Parrot frame. I don't understand what they are trying to do. There is ODID data in a bloated, non-broadcast frame and transmitting on a random channel channel is strange.

Anyway, I'm out re beacon. It doesn't seem the right way to go.

[gabrielcox]

@pierreSCpapon i analysed your captures with the OpenID dissectors and the WIFI Beacon protocol messages were found correctly. You captured a lot of wifi traffic so I had to use filter to select just broadcasts from the parrot. I was expecting to find them just in channel 6 but the situation is different. I need to check better the specs. In any case here below an example screenshot for both of your two cap files.

I noticed in the data you used a Parrot ANAFI. Do you know if the feature is available in all the versions of ANAFI?

@gabrielcox good to know you are working on the dissector to fix the header size problem. Let me know if you want to compare the updates.

I just committed the update (of the wireshark dissector) to main for the header size problem.
opendroneid/wireshark-dissector@26e1aaf

Feel free to create a PR if you have other changes you'd like mainlined.

Gabriel.

[pierreSCpapon]

It is great to see everyone collaborating in trying to crack this!
As requested here is a capture of the hostapd, ESP32 and Parrot ANAFI broadcasting simultaneously.

Thank you for the changes made to the dissector @gabrielcox I can now see the content of the ESP32 packets and hostapd however, oddly not the ANAFI even though you posted a screenshot of it working on your end... odd.
Could you give a got at dissecting this capture also?

parrotESPlinuxFilter.zip

[gabrielcox]

Thank you for the changes made to the dissector @gabrielcox I can now see the content of the ESP32 packets and hostapd however, oddly not the ANAFI even though you posted a screenshot of it working on your end... odd. Could you give a got at dissecting this capture also?

I think the example I have posted in the past was sending the Parrot OUI, but not an actual Parrot product.

I looked at the attached pcap (thank you).
If you're referring to the packets coming from the MAC addresses starting with Parrot_5b:, the problem is that they are not "Beacon" frames. They are "Probe Response" frames.

Here's what the standard requires for Remote ID - Wi-Fi Beacon:

5.4.9.2 For UAS implementing this protocol, a standard 802.11 Wi-Fi Beacon frame which is a “management” (type 0), “beacon” (subtype 8), frame as prescribed by the IEEE 802.11-2016 Part 11 Wi-Fi specification shall (BWFB0020) encode the Remote ID message pack as a vendor specific information element IE221 (using OUI: FA-0B-BC, Vendor Type: 0x0D) payload as described in Table 18.1.

The offset for this frame type is after the header, so it's typically at 0x12 (if the header length is 18 bytes). So, the correct frame type of the frame you currently have highlighted in the screencap properly shows 0x80 0x00 at offset 0x12 (which is why it shows "IEEE 802.11 Beacon Frame")

On the other frames (which are not broadcast frames since they have destination mac addresses), the frame types appear to be "Probe Response" frames (notice the 0x05 at offset 0x12).

Line 408 in the dissector will take you close to the logic where the filtering happens.

[friissoren]

Thanks @pierreSCpapon, for the captures. When looking at those, I started wondering if the discussion above about beacon and probe-response frames is also the reason behind the poor reception rate for the ESP32 transmitter data.

If you filter with wlan.tag.oui == 0xfa0bbc. There are only three beacon frames from the ESP32, 27 beacon frames from the Alfa hostapd transmitter and 6 beacon frames from the Parrot drone.
However, the probe response frames from both the hostapd and Parrot transmitters all contain also the drone ID data element and there is a significantly larger amount of these frames.

You have been capturing for around 10 minutes (as far as I could see). The first ESP32 ping is halfway during the capture so maybe you turned that one on later or it simply just failed to pick it up earlier than that. There seems to be significant gaps/jump in the captured data. I suspect the Wi-Fi receiver either goes to sleep mode once in a while or it scans on channels that are not having any transmissions on them related to drone ID?

Is it possible that the (phone) receivers pick up the drone ID from both the beacon and from the probe response frames? I.e. one way of improving the reception rate would be figuring out how to get the ESP32 to send probe response frames containing the drone ID data element?

This is a bit worrying though, since in a real-life situation, it is unlikely that the phones would be able to transmit a probe signal to the transmitter due to the distance between them and the (assumedly) low transmit power of the phone.

It is quite possible I am misunderstanding something here. This has never really been my area of expertise. But if the above is true, it doesn't put the Beacon transmission method in a very good position since relying on just the beacon frames clearly will lead to a very poor reception rate.

When looking at the used channels, the alfa transmitters uses 3,4,5,6,7,8 and 10. The ESP32 channels 5,6 and 7. And the Parrot use only channel 11.

[gabrielcox]

@friissoren, Pierre moved the issue to dissector repo (thanks). He also pointed out that I was looking at the wrong frames -- there are Beacon frames coming from the Parrot MAC in addition to the probe responses. I would expect the probe responses to simply be ignored (which is good) by the Android Receiver for 2 reasons. 1. They're not broadcast frames, 2. They're not Beacons (so the app should ignore as well). Since they're not broadcast, unless the destination MAC is the Android device, the Wireless NIC will just drop the frames and the OS will not see them. It will (/should) only let the broadcast and unicast frames directed to the Android MAC to come into the software layer(s). The exception is when in monitor/promiscuous mode (which the Android device shouldn't be in).

However, it does look like the "kitchen sink" is unnecessarily being sent in all frames. Other than general traffic pollution, I don't think it should be an issue.

Since the payload is also sent in beacon frames, I'll go troubleshoot that (and provide updates in the new issue Pierre opened up in the dissector repo. I think I already see the problem/bug in the dissector and the way the tags are being iterated.

[friissoren]

@gabrielcox: The main problem being investigated in this issue, is the very poor reception rate on some Android phones when transmitting Wi-Fi beacon signals on the ESP32. Neither the hostapd nor the Parrot Wi-Fi beacon transmitters exhibit this problem.

I need to do a test with the hostapd transmitter. The capture dump from Pierre showed that at least on his PC, the hostapd probe response frames had two vendor elements: drone ID and a Realtek one. If I in the Android application see the second one (or if there is a way to dig out from the data that it is a probe response) I can test on both the Samsung Galaxy S10 and the OnePlus 6T. I suspect that the S10 Wi-Fi driver will return the probe response data to the application but the OnePlus 6T Wi-Fi driver will not. Let's see.

[friissoren]

I tried to look through all the vendor specific elements that the Android app picks up on the S10 but when I run the hostapd transmitter on my Ubuntu desktop, only one vendor element is seen. That doesn't really prove anything though, since it is possible that my desktop Wi-Fi HW only send one (the ASD-STAN one).

I cannot get Wireshark to work in monitor mode with the HW I currently have. I have ordered a Wi-Fi dongle that hopefully will work, but it will probably take a while to get here from China. Need to put this investigation on hold, until then.

[BluemarkInnovations]

I have a background in WiFi analytics i.e. measuring footfall traffic by measuring WiFi signals from smartphones. I hope I can contribute to this issue by sharing my knowledge on this matter.

There are two methods, devices can detect nearby access points. a) passive scanning (listen to beacon frames) b) active scanning by sending probe request packets. Nearby Access Point respond with a probe response packet. Smartphones prefer method b) active scanning as it is much faster. I guess for passive scanning you need to listen say 250 ms per channel. (Default beacon rate is 100 ms for an AP.) There are 13 WiFi channels in the 2.4 GHz band and much more in the 5 GHz band. Hence, passive scanning would take a long time. I guess for active scanning less than 25 ms per channel would be required. Also if a smartphone is connected to a 5 GHz AP, some WiFi radios won't scan 2.4 GHz channels at all or less frequent.

So a smartphone does mainly active scanning, but I guess it will listen to beacon frames too if it receives one. Perhaps it also depends on the power saving configuration and radio hardware. Newer phones like to shut down WiFi radio as soon as possible to increase battery life. In the scanning results of nearby APs, you can't distinguish how they were detected (via probe response or beacon frame).

In my opinion, the beacon transmission protocol is not suitable to use in smartphone apps. My experience is that the performance varies a lot and is unreliable. It depends a lot on the scanning strategy of the phone.

Add probe request/response capability to the remote ID standards could be an option, but it is not really broadcasting. Potentially it could make such transponders vulnerable for attacks.