2 Omnipod Eros deactivated with 0x12 in Loop 3.2.3 even though Loop Cycle Time is limited

[philippfo9]

Describe the bug
After around 2 days, my Pod deactivates running a Dexcom G6 with Omnipod Eros and Loop 3.2.3.
Similar problem as here: #1924

Attach an Issue Report
Loop Report 2024-01-02 19:08:52+01:00.md

What could I do to avoid this?
Should I Limit Loop Cycle Time for a Dexcom G6 as well?

[philippfo9]

A week ago, I tried limiting the Loop Cycle Time to 5mins with the suggested patch.

Unfortunately, I am still experiencing the same issue, every Pod only runs for around 1d 20hrs - 2 days.

Loop Report 2024-01-30 11:21:21+01:00.md

Any suggestions? I am considering downgrading to Loop 2.

[marionbarker]

I just analyzed your two reports and there were a lot of communication issues.

Do you have a different RileyLink Device you can try?
Do you often leave your RileyLink out of range?

• First report
  • Pod 1: 12 nonce resync, 2 ACK, 0x34 @ 48.7 hours, #sent/#recv = ( 811 / 280)
  • Pod 2: 21 nonce resync, 1 ACK, 0x12 @ 48.4 hours, #sent/#recv = (1361 / 703)
• Second report:
  • Pod 1: 15 nonce resync, 1 ACK, 0x12 @ 65.2 hours, #sent/#recv = (1363 / 493)
  • Pod 2: 4 nonce resync, 1 ACK, still operating, #sent/#recv = ( 973 / 263)

Discussion:

• A nonce resync means the pod and loop have gotten out of phase (messages are being missed)
  • This can be a sign that the rileylink device and pod do not have a good connection
• The number of messages sent (from Loop) should be close to the number of replies (from the pod) received
  • Less than 40% of the messages sent from Loop got a reply from the pod
  • Doing a spot check in one log, I see a lot of noResponse from pod with CGM values coming in so phone is close enough to read CGM
  • If you keep your gear together (phone and rileylink near your body at all times), this lack of response to messages from Loop is another sign of communication issues

[philippfo9]

I don't have a different RileyLink Device right now.

In the nighttime, I keep the Rileylink on the bedside table where it doesn't bolus sometimes, so seems out of range - I always had the feeling that CGM due to just being Bluetooth has a better range.

During the day, I keep the RileyLink in my pocket all the time, but leave my phone on the bedside tables sometimes so I don't touch it while working from home.

Regardless of the different conditions, I never had this issue on Loop V2, but experienced it every time since Loop V3.
Has anything changed for V3 to ping the pod more often if a "nonce resync" happens?

The console is showing that the Limit CGM driven Loop cycle time is set to 5 minutes

Btw in my settings, I see that the pod has a different time configured, I just clicked to sync it. Could this trigger any of the issues above?

[marionbarker]

The limit cycle is for libre that reported every minute. Irrelevant for your situation.

Try keeping your rileylink close to the pod at all times. And look into getting a replacement. Open it and make sure antenna is securely attached. (Phone to riley is Bluetooth. Riley to pod is 533 MHz radio frequency and that’s probably what is failing.)

We recently discovered the uncertain comms handling for Eros in Loop 3 has a bug. (See Issue 2118). (Uncertain comms is probably what is going on with you. Loop 3 has a new method for handling that. )

Can you send me a PM in zulipchat and I’ll walk you through a rebuild of Loop using dev and show you how to build with the fix. It was just merged into OmniKit main branch (Eros pod code) but has not yet been added to LoopWorkspace dev branch.

[marionbarker]

Since we are 9 hours apart in time-zones, I wrote up the instructions. If you do not feel confident with these let me know. I assume you are using a Mac to build.

Download these 2 files.

• 20240130_Eros_Fix.txt
• OmniKit-24.patch

Open the 20240130_ErosFix.txt file in the TextEdit app (or other text-only app). There are commands for you to copy and paste into the terminal that have double hyphens, i.e., --. These get messed up if you open the file with a smart editor.

The instructions in the txt file tell you how to get all the fixes that are already merged into OmniKit and how to use the patch file for the final fix that is not yet merged.

If you are successful with the build, please post another log file after 24 hours of operation to see if this improves things. And if you are near the end of life of a given pod, don't expect miracles. Hope this will help with the next pod.

[philippfo9]

Hey, thanks a lot for your help!

Yep opened up the .txt file in VS Code, I definitely feel confident with the instructions, I'll will rebuild it and report back.

[philippfo9]

Hey @marionbarker I think this has improved my issue, thanks a lot for your help so far!

The first pod (1 day old) has run out after 1 day 22 hours.
The second pod managed to do 2 days 22 hours (not the full 3 days + 8 hours, but better).
The third one is still running and has already crossed 2 days & 10 hours.

I also ordered a new OrangeLink. I hope I can manage to run 3 days + 8 hours again without issues then.

Here is the latest report:
Loop Report 2024-02-13 16:16:16+01:00.md

[marionbarker]

Thank you for the report. This shows definite improvement in recovery from uncertain communications.

The characteristic of uncertain comms is a request for update at about 60 s intervals until the problem is resolved.
Without the patch, the recovery took a long time after the first response from the pod, now it should recover as soon as a response comes back.

I counted the number of messages where Loop sent another message between 45 and 75 sec - called this repeated pings. (Not a perfect measure, but an easy one.) I then divided by hours in the record to get an estimate of frequency.

There are 2 pods in this report:

• pod 1: Fault 0x12 at 68.2 hour
  • Final 27 hours of pod life captured in the log
  • Number of messages, captured in the log, sent by loop vs response from pod: 701 / 351
  • Repeated pings ~196; average 7 per hour
• pod 2:
  • Still operating at 55.1 hours (full life history in log)
  • Number of messages, full life of pod, sent by loop vs response from pod: 730 / 576
  • Repeated pings ~64; average 1 per hour

They both show symptoms of poor communication, but with the "bug" fixed in the code, this should cause less of a load on your pods. I hope the newer OrangeLink reduces the uncertain comms events and that will help as well.

I went back to an older report from early January

• Example pod:
  • Fault 0x12 at 49.3 hours
  • Repeated pings: 372; average of 7 per hour

[marionbarker]

@philippfo9 Thank you so much for reporting.

The fixes that you applied with the patch information I sent you were merged into LoopWorkspace dev on 14 Feb 2024.

There is no need for you to rebuild at this time.

Once the new release comes out, this Issue can be closed. But in the meantime, please leave it open as the fix is in dev and not in main (right now).

[philippfo9]

Hey thanks @marionbarker!

OrangeLink arrived, will try it out soon :)

Just had the following error: 0x34 (reset from unknown cause) after just 36 hours, which might be unrelated.

Attaching the report
Loop Report 2024-02-15 23:14:01+01:00.md

[marionbarker]

Well, the good news is the first pod in the log made it to 72.8 hours before getting an 0x12 fault.

• messages (sent / recv) = ( 981 / 573)
• ping count was 220, or 3 per hour - so that's higher again

Unfortunately, that second pod did fail at 35.8 hours with a 0x34 fault.

• messages (sent / recv) = (1077 / 451)
• ping count was 436, or 12 per hour which is really high

I hope the new OrangeLink will help with the comms and stop these faults. I know how intensely annoying it is to get screamers.

Please send me logs again and indicate the time at which you switch to the new OrangeLink.
You can switch to the OrangeLink in the middle of a pod, no need to wait.

• https://loopkit.github.io/loopdocs/faqs/rileylink-faqs/#adding-or-changing-rileylink

[philippfo9]

Yes I agree, it's annoying, especially with the amount of Pods it's hard to justify replacements for social insurance.

Anyway, I just had the same issue again, the pod is running a total of ~44 hours with a 0x34 fault.

Just bought AAA batteries today for the OrangeLink and making the switch now, wish me luck! Reporting back here in around 24-48 hours :)

[philippfo9]

@marionbarker with OrangeLink, 3 days in and still running.

Report:
Loop Report 2024-02-20 21:57:35+01:00.md

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[marionbarker]

Bump. Fixed in dev. Leave open until released.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[marionbarker]

Bump