Allow external CodeInstances to be added to the execution engine

[Keno]

As of #35831, we've had the ability to cache CodeInstances in an
external cache and to replace that cache during jl_generate_native
(e.g. for GPU compilation). This extends the same capability to
CodeInstances to be added to the execution engine.

[vtjnash]

It seems like this needs to be creating a separate instance of our internal JIT customized on the cg_params to avoid polluting the normal cache. As it is, I wanted to hide this method from users and then eventually hope to reduce it in scope and capability (It currently exists to help handle the transitional boundary between the internal statefulness of the JIT and the external statelessness of our callable/interpreter semantics. So this function felt like an awkward mix of policies and implementations, which is possibly just working around deficiencies in both right now.)

[Keno]

Which internal cache are you thinking of that would be polluted? The CodeInstances I'm imagining passing here would not be part of the regular cache structure. Is there another cache I should be looking at?

[Keno]

@vtjnash ping. Could you elaborate on which caches you mean?

[Keno]

Rebased.

[vtjnash]

This just seems like it could greatly complicated and/or make invalid various other promises in the system through negative interactions with the expectations of this cache (such as incremental compilation and the compile=all flag). There's a reason the external entry points try to work with MethodInstances and hide these objects. Even their presence as an argument here I think might be a bit of an accident: we optionally pass it along to avoid duplicate work (whether a cache re-lookup or re-inference), but it's not entirely straightforward whether we promise to preserve that information. We sometimes also may discover it necessary to inject the result back into the normal cache under a different key. Currently we just ignore bugs that could arise from that situation. This method also sometimes fails for other reasons, which the caller must handle.

[Keno]

Can you clarify which cache you're talking about? Are you talking about he various fields inside the CodeInstance?

[vtjnash]

No, mostly we consider the fields to be write-once so that there's no mutation issues. But hence why we may want to be able to interact with the various other caches in the system if the result can't fit into the input.

[Keno]

Could you please be more precise as to which caches you're thinking of? I was under the impression that at this point the only relevant caches are those inside the code instance objects (potentially returned via the lookup function). If there's any other caches, I obviously need to address those. I really want to be able to have a totally separate set of caches that I can manually manage, so I can do horrible things to the code without worrying about corrupting anything in the main cache.

[vtjnash]

Like Method, MethodInstance, and Module, while they in theory can be made to exist outside of the normal structure, some parts of the system are designed to expect that they are not permitted to be.

That said, upon reflection, we do have an existing mechanism for that purpose. We essentially define that eval is the entry point for generally doing horrible things, and then pass it a thunk (basically a CodeInstance wrapped in some metadata) and hope that it knows horrible things are going on and to try not to get in the way too much of that, but to just trust that it'll work out okay eventually.

[Keno]

I was really hoping for a specific example of what would go wrong with the CodeInstance. I agree that this isn't a suitable long term solution (for all sorts of reason - e.g. it doesn't precompile properly), but it's useful for experimentation to just completely take a CodeInstance out of the cache hierarchy and be able to operate on it independently. I'm thinking that in the long term, these will probably just go back into the regular method cache, keyed by some context a la Cassette, perhaps with a way to attach specific cgparams to a CodeInfo, but that's a mechanism I don't want to design until I have more of an idea of all the things it needs to do, so these kinds experimental hooks are useful until then.