Cache MDX compiler in the Webpack loader

[remcohaszing]

A new MDX compiler was created for every file that’s processed. This means that any initialization logic in remark or rehype plugins was always run for every file.

By reusing the MDX compiler, remark and rehype plugins can run heavy setup logic once, just like when they are called by unified directly.

In practice I use this in remark-mermaidjs. This starts a browser once to process a bunch of files, but it requires the same unified processor to run to be able to do so.

The same principle is used in ts-loader to cache the TypeScript compiler.

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/mdx/mdx/98pc7aCniJSuaojuJsD8ikpgZopW
✅ Preview: Failed

[slorber]

@wooorm has this caching system been removed in v2?

I don't see it anymore here: https://github.com/mdx-js/mdx/blob/main/packages/loader/lib/index.js

[wooorm]

I believe xdm was fast enough, it does a lot less work. Do you need it then?

[remcohaszing]

The main issue isn’t the MDX compiler, but the plugins that may be slow.

Specifically remark-mermaidjs, which uses puppeteer, is slow. A lot of heavy lifting could be ran only once instead of per file when reusing the same browser instance, but this requires the unified pipeline to be reused.

Personally I haven’t been affected by this issue lately, because I disabled remark-mermaidjs in development (watch) mode. Still this optimization would be nice for those who do use it in development, and it would also significantly speed up production builds for my project.

[wooorm]

Oh right. This was around the xdm time, I believe the plan was to land it there too but it never did?
xdm has proper caching mechanisms for rollup, esbuild, and node, which have been ported over to mdx-js/mdx. But not for webpack.

[wooorm]

added again!