Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GM: ACC faults #24223

Closed
adeebshihadeh opened this issue Apr 16, 2022 · 16 comments
Closed

GM: ACC faults #24223

adeebshihadeh opened this issue Apr 16, 2022 · 16 comments
Assignees
Labels
bug car vehicle-specific

Comments

@adeebshihadeh
Copy link
Contributor

Events while engaged on 0.8.13-release

CHEVROLET VOLT PREMIER 2017              85a49685933befe2|2022-02-24--15-22-05--5 7
CHEVROLET VOLT PREMIER 2017              1f0e64fd786bf5ed|2022-02-26--15-42-18--1 6
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-02-26--16-38-50--10 7
CHEVROLET VOLT PREMIER 2017              1f0e64fd786bf5ed|2022-03-01--14-09-07--2 6
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-01--14-42-51--35 6
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-02--17-15-36--34 7
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-02--18-44-06--1 7
CHEVROLET VOLT PREMIER 2017              d2ccdcf33e62cfa0|2022-03-04--13-58-06--8 7
CHEVROLET VOLT PREMIER 2017              3954275d7161f9a8|2022-03-05--11-36-56--17 6
CADILLAC ESCALADE ESV 2016               2e5410d8378a30c6|2022-03-08--18-09-17--20 6
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-08--22-47-34--20 6
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-10--00-25-40--1 6
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-10--08-38-12--26 7
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-10--08-38-12--27 7
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-10--08-38-12--28 7
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-10--08-38-12--29 7
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-11--13-25-56--4 6
CHEVROLET VOLT PREMIER 2017              d2ccdcf33e62cfa0|2022-03-12--19-23-43--2 7
CHEVROLET VOLT PREMIER 2017              9425b0775a4688a1|2022-03-12--11-42-35--74 6
CHEVROLET VOLT PREMIER 2017              87b41dfea45f3cce|2022-03-12--13-49-33--96 6
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-13--18-36-37--9 7
CADILLAC ESCALADE ESV 2016               495552ac53209990|2022-03-13--18-54-37--12 7
CHEVROLET VOLT PREMIER 2017              d2ccdcf33e62cfa0|2022-03-13--18-50-46--3 7
CADILLAC ESCALADE ESV 2016               87b41dfea45f3cce|2022-03-16--19-32-51--4 6
CHEVROLET VOLT PREMIER 2017              87b41dfea45f3cce|2022-03-17--14-26-47--12 6
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-18--22-16-59--2 6
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-18--22-16-59--26 6
CHEVROLET VOLT PREMIER 2017              bfe8da33a68f85f2|2022-03-18--20-18-29--13 6
GMC ACADIA DENALI 2018                   2cb99e3ec962642f|2022-03-19--14-13-59--7 7
CHEVROLET VOLT PREMIER 2017              bfe8da33a68f85f2|2022-03-19--13-02-59--31 6
CHEVROLET VOLT PREMIER 2017              a1e78442f3848534|2022-03-20--14-08-49--13 6
CHEVROLET VOLT PREMIER 2017              9425b0775a4688a1|2022-03-20--10-25-05--64 6
CHEVROLET VOLT PREMIER 2017              bfe8da33a68f85f2|2022-03-21--12-45-46--23 6
CHEVROLET VOLT PREMIER 2017              bfe8da33a68f85f2|2022-03-21--13-18-01--55 6
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-03-22--16-22-33--50 7
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-03-22--16-22-33--79 7
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-23--14-18-49--5 6
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-24--22-50-30--65 6
GMC ACADIA DENALI 2018                   2cb99e3ec962642f|2022-03-27--11-17-29--11 7
GMC ACADIA DENALI 2018                   2cb99e3ec962642f|2022-03-27--21-01-44--138 7
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-03-30--21-51-40--37 6
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-04-01--12-23-47--1 7
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-04-01--13-03-07--3 7
CHEVROLET VOLT PREMIER 2017              139dc5d40eb1be7c|2022-04-02--18-11-55--21 7
GMC ACADIA DENALI 2018                   2cb99e3ec962642f|2022-04-02--18-45-14--9 7
CHEVROLET VOLT PREMIER 2017              bfe8da33a68f85f2|2022-04-03--19-18-53--8 6
CHEVROLET VOLT PREMIER 2017              590fd6ee1f594beb|2022-04-04--07-56-57--8 6
CHEVROLET VOLT PREMIER 2017              590fd6ee1f594beb|2022-04-05--07-50-52--6 6
CHEVROLET VOLT PREMIER 2017              a3e8ad84037083ca|2022-04-05--14-40-54--7 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-05--20-35-08--2 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-05--20-35-08--3 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-05--20-35-08--5 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-05--20-42-32--1 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-06--00-25-59--3 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-06--00-25-59--9 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-06--00-25-59--11 7
CHEVROLET VOLT PREMIER 2017              a3e8ad84037083ca|2022-04-06--08-02-45--2 7
CHEVROLET VOLT PREMIER 2017              20cb801701fc8237|2022-04-06--13-09-20--8 7
CHEVROLET VOLT PREMIER 2017              2e5410d8378a30c6|2022-04-06--20-52-23--29 6
CHEVROLET VOLT PREMIER 2017              3954275d7161f9a8|2022-04-06--13-04-46--3 6
CHEVROLET VOLT PREMIER 2017              d2ccdcf33e62cfa0|2022-04-07--12-44-16--4 7
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-04-08--18-19-25--4 7
CHEVROLET VOLT PREMIER 2017              87b41dfea45f3cce|2022-04-11--07-45-46--5 6
CHEVROLET VOLT PREMIER 2017              87b41dfea45f3cce|2022-04-11--07-45-46--8 6
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-04-12--09-26-50--206 7
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-04-12--15-55-15--51 7
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--08-36-57--4 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--08-36-57--5 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--08-36-57--19 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--08-36-57--20 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--08-36-57--30 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--08-36-57--37 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--12-29-10--5 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--12-29-10--17 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-13--14-49-09--8 6
CHEVROLET VOLT PREMIER 2017              e2b25f39da040caa|2022-04-13--16-11-40--38 7
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--08-29-47--15 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--08-29-47--16 6
CHEVROLET VOLT PREMIER 2017              6363a0e48f515f27|2022-04-11--21-09-30--3 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--14-24-56--6 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--14-24-56--11 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--15-28-48--12 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--15-28-48--16 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--15-28-48--21 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--16-50-16--4 6
CHEVROLET VOLT PREMIER 2017              3e336a3be4261460|2022-04-14--18-23-56--35 6
CADILLAC ESCALADE ESV 2016               3e336a3be4261460|2022-04-15--09-57-35--4 6
CADILLAC ESCALADE ESV 2016               3e336a3be4261460|2022-04-15--09-57-35--17 6
@hewers
Copy link
Contributor

hewers commented Apr 17, 2022

Diagnosed my Volt ACC faults to a known brake-pressed-sensor vehicle issue.

  • Symptoms
    • Cruise fault on engage, "repeatedly fails to engage"
    • Cruise fault on road bumps
    • Cruise fault on tapping brakes

Narrative: vehicle ACC and openpilot are using different brake-pressed signals. Vehicle is more sensitive, and wants to disengage but openpilot continues with ACC commands. In 0.5 seconds, vehicle ACC faults.

  • TODO
    • Find the correct ACC brake-pressed disengage signal
    • Brake pedal failure behavior should be: soft-disable "Brake pedal pressed"
    • Clean up documentation on this vehicle brake pedal failure, users may experience unexpected brake pedal disengagements. Mitigation is that users are always ready to resume control.

@hewers
Copy link
Contributor

hewers commented Apr 19, 2022

image
New brake pressed from same message as brake pedal position, will backtest.

@Verylukyguy
Copy link
Contributor

I have been getting intermittent RSOD CAN errors when I press the brake when OpenPilot is engaged, but they resolve immediately. This just started happening, maybe since the GM brake signal was changed.

@sshane
Copy link
Contributor

sshane commented May 7, 2022

This PR is causing ACC faults on cars that have a noisy brake pedal: #23712

The car's ACC system actually uses the noisy ECMAcceleratorPos->BrakePedalPos signal with a threshold of 8, and whenever it rises above that, the ACC system wants to disengage. openpilot knows the user didn't actually press the brake, so it stays engaged causing an ACC fault.

Ideally we should detect this and alert the user when we disengage to prevent a fault.

Looked into if there were any bits that describe that threshold, but there are none.

Plot showing a fault occurs shortly after that signal goes to 8, while the filtered/true brake signal (that we use now) stays zero:

Screenshot from 2022-05-07 00-19-42

@hewers
Copy link
Contributor

hewers commented May 7, 2022

https://github.com/qadmus/openpilot/tree/gm-fix-acc-fault-brake-position

Worked this morning with my light-brake-press test method.

Traveling this weekend, will make PR if no one gets to it first.

@sshane
Copy link
Contributor

sshane commented May 21, 2022

Looked at all the segments here that we have rlogs available for, most don't seem to be caused by any brake pedal issue:

Total segments with faults: 27, would be able to prevent by disengaging with brake pedal: 6
Reverted PR would have caught: 5  # https://github.com/commaai/openpilot/pull/23712
Master would have caught:      0
New PR would have caught:      6  # https://github.com/commaai/openpilot/pull/24589

@JMPZ11
Copy link
Contributor

JMPZ11 commented May 21, 2022

Guys - the 8 threshold is not the same on all cars. The switch away from it originated when I needed to bump the threshold to 11 to accommodate trucks - The suburban, Silverado and Tahoe were getting random disengagements.

Adeeb requested finding and using a boolean signal, - both Jason Young and someone at comma confirmed the validity of the brake-pressed signal against lots of data.

@sshane - it was my understanding that the stock ACC is controlled by the ASCM, and that the ASCM must be disconnected for all currently supported GM vehicles. The ASCM sends gas and brake commands directly to the PT / Chassis busses.

OP completely replaces the functionality of the ASCM - where is this disengagement coming from?

We have an experimental VOACC branch that is able to control gas/brake without syncing any sort of ACC state. When the gas command is active, the car applies gas, etc.

@hewers
Copy link
Contributor

hewers commented May 22, 2022

Note PR #24589 is a new source of Brake Pedal Position, from ECMAcceleratorPos. The signals and subsequent thresholds are different than other brake position sources.

@JasonJShuler I'd guess the ECM disengages, as a redundant test of the brake pedal position. I don't trust the names of these messages though, don't know where they came from.

We may need BOTH (and more) brake pressed detections. And if models or worse, individual cars, have calibrated thresholds, which is possible, then finding a boolean signal is really valuable.

@sshane
Copy link
Contributor

sshane commented May 22, 2022

@JasonJShuler I'm not convinced all these faults are caused by the brake pedal issue, it's just a few could have been prevented and we wanted to revert back to a state where we would have caught at least some faults. If you believe different cars use different brake pressed signals/thresholds we should make a safety param flag for that based on the car. We should also figure out what is causing these faults in the process. I checked and blocked messages are also not the cause.

Yeah there must be a redundant check in the ECM (/PCM, or is that the ASCM?) where it won't allow you apply brake or gas if the user is pressing the brake, which it thinks they are based on the noisy signal we removed.

@adeebshihadeh
Copy link
Contributor Author

Guys - the 8 threshold is not the same on all cars. The switch away from it originated when I needed to bump the threshold to 11 to accommodate trucks - The suburban, Silverado and Tahoe were getting random disengagements.

We simply reverted back to avoid regressions for currently supported cars. If these new cars already kind of work, it'll be easier to find a solution if you open a PR and include some routes.

@sshane
Copy link
Contributor

sshane commented Aug 18, 2022

Quickly estimated how many of these faults would be fixed by #25402. Seems like a pretty sizable chunk of the segments with rlogs. I can likely adapt to work on qlogs later.

The criteria was engaged shorter than 600 ms ago with the resume button (since we can be in any part of the route we don't know if we ever engaged after main toggle before this (can be solved with looking at the whole qlog route)):

Total segments with faults: 27
Likely caused by resume bug: 13

@sshane
Copy link
Contributor

sshane commented Oct 15, 2022

Camera ACC ACC faults:

@adeebshihadeh
Copy link
Contributor Author

@sshane ready to close or are we still working through some?

@sshane
Copy link
Contributor

sshane commented Nov 10, 2022

There's still a lot I don't understand from this list. I may have figured out a sizable chunk of the remaining camera ACC faults, so I'll re-check all the ones above with rlogs

@adeebshihadeh adeebshihadeh removed this from the 0.8.17 milestone Nov 11, 2022
@sshane sshane closed this as completed Feb 10, 2023
@sshane sshane reopened this Feb 10, 2023
@sshane
Copy link
Contributor

sshane commented Feb 10, 2023

Bolt EUV camera ACC faults:

  • Enabling at a stop shortly after reversing -- the first enable will result in an ACC fault
  • After rebooting the device while car is on (you can do this without an LKAS fault once or twice), we lost brake control and the car kept coasting into the lead: 6e997623cb7cdaa6|2023-02-09--16-19-40--0 We check the ECM for cruise faults, but I guess the brake controller must have its own signal somewhere
  • Unknown ACC fault in the Bolt: 6e997623cb7cdaa6|2023-02-09--16-20-24--1

@sshane sshane closed this as completed Mar 25, 2023
@sshane
Copy link
Contributor

sshane commented Mar 25, 2023

Closing since we've now solved most faults (if not all) in the initial list

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug car vehicle-specific
Projects
None yet
Development

No branches or pull requests

6 participants