Loopback application middleware not working - No matching layer found errors

[prithviraj99]

I have integrated my loopback application with appDynamics library and all the middlewares are not firing from then on.
Used the debug command - DEBUG=* npm start to get the logs and found these logs coming up.
All the middlewares declared through middleware.json and imperatively through app.middleware() command are having this error message - No matching layer found

2020-07-10T15:23:11.241Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:app No matching layer is found for initial:before favicon
2020-07-10T15:23:11.241Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:boot:executor Configuring middleware "/usr/nodejs/njsDT_SIT/apps/wlpn/ob-cof-funds-confirmation-consents-channel/package/node_modules/compression"
2020-07-10T15:23:11.243Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:app use initial  compression
2020-07-10T15:23:11.244Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT express:router use [] compression
2020-07-10T15:23:11.245Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT express:router:layer new []
2020-07-10T15:23:11.245Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:app No matching layer is found for initial compression
2020-07-10T15:23:11.245Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:boot:executor Configuring middleware "/usr/nodejs/njsDT_SIT/apps/wlpn/ob-cof-funds-confirmation-consents-channel/package/node_modules/loopback-context/server/middleware/per-request.js"
2020-07-10T15:23:11.246Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:app use initial  perRequestContext
2020-07-10T15:23:11.253Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT express:router use [] perRequestContext
2020-07-10T15:23:11.254Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT express:router:layer new []
2020-07-10T15:23:11.254Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:app No matching layer is found for initial perRequestContext
2020-07-10T15:23:11.254Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:boot:executor Configuring middleware "/usr/nodejs/njsDT_SIT/apps/wlpn/ob-cof-funds-confirmation-consents-channel/package/node_modules/cors"
2020-07-10T15:23:11.254Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:app use initial  corsMiddleware
2020-07-10T15:23:11.254Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT express:router use [] corsMiddleware
2020-07-10T15:23:11.254Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT express:router:layer new []
2020-07-10T15:23:11.254Z - : pid:36871 worker:1 Fri, 10 Jul 2020 15:23:11 GMT loopback:app No matching layer is found for initial corsMiddleware

Any idea what's going wrong here - Which loopback package is responsible for express routing implementation

[dhmlau]

@hacksparrow, could you please take a look? Thanks.

[hacksparrow]

@prithviraj99 does it work fine without appDynamics integration?

[hacksparrow]

This file is where the middleware stuff is implemented - https://github.com/strongloop/loopback/blob/master/lib/server-app.js.

[bajtos]

Thank you @prithviraj99 for reporting the issue.

In order to support middleware phases, LoopBack has to monkey-patch internal data structures used by Express to hold routes. Many monitoring agents (e.g. NewRelic) monkey-patch these internal structures too. Unsurprisingly, when two parties are patching the same place, a conflict can occur. In the past, we have fixed our middleware phases to recognize NewRelic patches - see #1131. Perhaps we need to implement a similar workaround to support appDynamics?

[prithviraj99]

@bajtos @hacksparrow @dhmlau ,

Sorry for the delay but this is the update i have -

"I have identified the trouble caused by AppDynamics on application code (on loopback’s middleware router_stack) and understood that this was indeed fixed on the version of loopback v3.25.0 onwards. https://github.com/strongloop/loopback/blob/v3.25.0/lib/server-app.js

This is the commit on Loopback’s official github page - edb8dbc

Commit description - AppDynamics injects a proxy object into the router stack, which it uses for its network analysis. This is similar to how NewRelic adds a sentinel handler to the router stack. This commit adds a similar workaround so that loopback can find the original layer.

Our service was having a version of loopback v3.12.0 and hence this problem existed in first place. I have fixed it by upgrading to higher 3.x version."

[bajtos]

Awesome, thank you for confirming the fix works for you!

I am closing this issue as fixed via #4119 then.