Support Substack

[gustavo-depaula]

The extension is great for doing code blocks in Medium. Any thoughts on supporting Substack, or do you know a similar solution to substack?

[Maluen]

No plans on supporting Substack for now, as this extension was designed and built with medium only in mind. I've never used Substack myself so first we would indeed have to look at the current solutions and at the editor itself.

A main requirement is for them to allow gist embedding. From a quick search looks like it isn't possible?

[gustavo-depaula]

@Maluen it is possible:

But IDK why this text about bidirectional unicode text happens

[Maluen]

The unicode warning is specific to gist, see https://github.blog/changelog/2021-10-31-warning-about-bidirectional-unicode-text/ I suggest trying with other gists, e.g. by creating one from scratch.

Substack supporting gist is a good first step as we can reuse all the background part of the extension and also all the UI for creating/updating gists/logging-in.

So the part still left to implement would be extending their WYSIWYG editor:

• button to show the "create new gist UI"
• double click on existing gists to edit them
• CSS to make sure the extension UI doesn't break their editor.

[gustavo-depaula]

Hey! I was able to create a new service to show the button:

But for some reason the event listeners don't seem to work inside the dropdown... Even when using window.addEventListener("click", console.log), clicks on the dropdown items are not triggering clicks. Maybe there are some event.stopPropagation/preventDefault that are "swallowing" these events?

Here's how I'm creating the button: gustavo-depaula@bf3a063#diff-32bbb3346e15f8054cffefe0a61d4a814a64dcf6d61674aeda6795c56ce21c6bR68-R82

[Maluen]

Hey! I was able to create a new service to show the button:

Nice work! Looks promising! I think in terms of CSS we'll have to separate the medium styles from the substack styles to limit regressions. E.g. by using a different CSS file for each (with some common imported styles if needed, better if not).

Maybe there are some event.stopPropagation/preventDefault that are "swallowing" these events?

I'll have to check on the actual website. The last resort is window.addEventListener("click", handler, true) (notice the third argument being true), where the handler function checks if event.target is the button or an element inside the button.

You can first try passing true when attaching the event to the element itself buttonEl.addEventListener('click', this.handleCreateGistClick, true);

[gustavo-depaula]

I think in terms of CSS we'll have to separate the medium styles from the substack styles to limit regressions.

Absolutely! I'm just trying to get it to work before doing all these design/architectural decisions.

You can first try passing true when attaching the event to the element itself

It didn't work... Might it perhaps be because it's not button?

[Maluen]

I've checked the substack editor. The dropdown closes on mousedown, thus the click never registers. Code below works:

buttonEl.addEventListener('mousedown', this.handleCreateGistClick);

The handleCreateGistClick needs to be adapted for substack as with other functions.

[Maluen]

See #12 and 465f7c2

I've also changed insertGistIntoPost to simulate a paste event with the gist URL.

For gist editing, I've noticed that they are not embedding the IFRAME with the gist, they are actually pulling the HTML and adding it inside a regular DIV. This means a different method needs to be used to update the gist.

[Maluen]

3e10589

• Added dblclick handler to trigger edit UI when double clicking on gists.
• Added code to delete a gist (`deleteGistIntoPost')

The main thing still left to do in SubstackService is the updateGistIntoPost function, which is supposed to refresh a gist with its new version after the editing is finished. For now I've tried deleting and then repasting the gist URL, but for some reasons the editor reads it as a plain URL and doesn't convert it into an embed. Will need to find a workaround.

[gustavo-depaula]

Nice!! Wouldn't have found the event stuff. Thanks

I think it might be a problem with the link itself. Pasting manually:

https://gist.github.com/gustavo-depaula/aeeccfa7056acf75836b57a23c063c89  -> this creates an embed
https://gist.github.com/aeeccfa7056acf75836b57a23c063c89 -> this doesn't

(the difference is the username, that must be simple to inject, gonna try it)

[gustavo-depaula]

done the username injection in gustavo-depaula@d46cd3f

insertion is now working as expected:

[gustavo-depaula]

Here's what I'm thinking about update: we could trigger a delete/backspace, that would delete the current embed, and then call the insert function again. That would fetch the new content

[gustavo-depaula]

Just noticed that there's already a simulateBackspaceKeydown function and that deleteGistIntoPost uses it.
Got the update working in by using the delete/insert-again approach: gustavo-depaula@3abf811

demo.mp4

Still being tormented by the bidirectional unicode text thingy, though...

[gustavo-depaula]

Thought in just deleting the warning by doing:

      const bidirectionalUnicodeTextWarn = document.querySelector('.flash-warn');
      bidirectionalUnicodeTextWarn.parentElement.removeChild(bidirectionalUnicodeTextWarn);

but (obviously) doesn't work because when it'll be published the warning will appear hahahaha

[Maluen]

Yes, the update wasn't working in my tests because of the different URL structure. Nice find on that!

About the unicode warning, that is indeed very annoying, I don't think we should add Substack support to the extension if every gist shows the warning.

I tried creating a new one-line gist directly from gist.github.com and then pasting it into Substack, it still shows the warning, which makes me think it might be related to how Substack is fetching the gist on their server-side. Maybe a bug on their end?

[gustavo-depaula]

Yes... It's my best theory also. Using the network request, we're able to see that the message is returned on the response of their gist embed API.

Thus, I have contacted substack support reporting this bug. Let's see what they say.

Aside from that, is there any feature/behavior left to implement?

[Maluen]

Thus, I have contacted substack support reporting this bug. Let's see what they say.

Cool, let's hope for a quick solution.

Aside from that, is there any feature/behavior left to implement?

Not much. There is code in SubstackService left from the MediumService to prevent interaction with the editor while the extension UI is visible (i.e. while gists are being created/edited): https://github.com/gustavo-depaula/code-medium/blob/3abf81181848b2fb481675d8028834a2769f752b/src/content/services/SubstackService.js#L72-L102

That is needed when the extension UI is displayed on the right side in big screens. In that case the user shouldn't be able to edit the article, but they should still be able to select text if possible. However even just allowing text selection could break the extension when clicking on "Create gists" since the focus could have been lost, the cursor have changed position, etc.

Other than that there is the CSS separation I talked before, where content.scss can be splitted into medium.scss and substack.scss.

I think I will also want to rename content_iframe.js into medium_iframe.js.

Basically making it clear what is medium and what is substack.

[Maluen]

Code restructured here: 0632335

[gustavo-depaula]

hey, closing this as I'm not using substack and never got an answer

thanks for your support, though! great extension

[KyMidd]

Did we ever receive an update from Substack on the unicode warnings? I'm also pushing for an answer, as I use gist extensively embedded in Medium pages, and as I'm moving to Substack I'm seeing all these warnings.

Thanks for all ya'll do.

[Maluen]

I'm still seeing unicode warnings for most gists on substack.

I just tried to create a new gist (directly from github) with only the word "test" inside, no spaces, no newlines, and substack is still showing the unicode warnings when embedding.

The same warning also shows on the published post, so it isn't just a draft/preview issue.

[KyMidd]

I don't work at github, but know some folks well who do. I'm escalating this with GitHub support.

I see the error message coming from github directly, here's a link to a gist with string "asdfasd" only, no returns or characters, not copied from anywhere, no invisible unicode characters, and it still shows the error.

I'll let you know when/if we get anywhere :) Thanks @Maluen!

[Maluen]

Indeed, Substack isn't embedding the gist as an iframe like Medium, it is pullilng the HTML itself from the gist URL, which contains the warning as visible in your link.

[KyMidd]

Oh, so this isn't a GitHub change. That's a bummer, I don't have any sway at Substack. I'm not very hopeful to evince changes over there. Okay, time to migrate all my (hundreds) of gists to direct embedded codeblocks. Thanks Maluen, I appreciate ya.

[Maluen]

Looking at your link and executing the JS (it's a bunch of document.write calls), we can see that the unicode warnings are actually inside template elements and aren't actually rendered.

The rendered page shows no warning

However the warnings are shown when embedded in Substack.

Looking at substack. they are doing an API call to api/v1/github/gist that returns the following JSON:

{
    "innerHTML": "<div id=\"gist131045874\" class=\"gist\">\n    <div class=\"gist-file\" translate=\"no\" data-color-mode=\"light\" data-light-theme=\"light\">\n      <div class=\"gist-data\">\n        <div class=\"js-gist-file-update-container js-task-list-container\">\n  <div id=\"file-asdf-tf\" class=\"file my-2\">\n    \n    <div itemprop=\"text\" class=\"Box-body p-0 blob-wrapper data type-hcl  \">\n\n        \n<div class=\"js-check-bidi js-blob-code-container blob-code-content\">\n\n  <template class=\"js-file-alert-template\">\n  <div data-view-component=\"true\" class=\"flash flash-warn flash-full d-flex flex-items-center\">\n  <svg aria-hidden=\"true\" height=\"16\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" data-view-component=\"true\" class=\"octicon octicon-alert\">\n    <path d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"></path>\n</svg>\n    <span>\n      This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.\n      <a class=\"Link--inTextBlock\" href=\"https://github.co/hiddenchars\" target=\"_blank\">Learn more about bidirectional Unicode characters</a>\n    </span>\n\n\n  <div data-view-component=\"true\" class=\"flash-action\">        <a href=\"{{ revealButtonHref }}\" data-view-component=\"true\" class=\"btn-sm btn\">    Show hidden characters\n</a>\n</div>\n</div></template>\n<template class=\"js-line-alert-template\">\n  <span aria-label=\"This line has hidden Unicode characters\" data-view-component=\"true\" class=\"line-alert tooltipped tooltipped-e\">\n    <svg aria-hidden=\"true\" height=\"16\" viewBox=\"0 0 16 16\" version=\"1.1\" width=\"16\" data-view-component=\"true\" class=\"octicon octicon-alert\">\n    <path d=\"M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z\"></path>\n</svg>\n</span></template>\n\n  <table data-hpc class=\"highlight tab-size js-file-line-container js-code-nav-container js-tagsearch-file\" data-tab-size=\"8\" data-paste-markdown-skip data-tagsearch-lang=\"HCL\" data-tagsearch-path=\"asdf.tf\">\n        <tr>\n          <td id=\"file-asdf-tf-L1\" class=\"blob-num js-line-number js-code-nav-line-number js-blob-rnum\" data-line-number=\"1\"></td>\n          <td id=\"file-asdf-tf-LC1\" class=\"blob-code blob-code-inner js-file-line\">asdfasd</td>\n        </tr>\n  </table>\n</div>\n\n\n    </div>\n\n  </div>\n</div>\n\n      </div>\n      <div class=\"gist-meta\">\n        <a href=\"https://gist.github.com/KyMidd/fb0cd10efd25909a66e7ff2016327d69/raw/4e956328ce4aa5e31c4ad5c0303903f84f340336/asdf.tf\" style=\"float:right\" class=\"Link--inTextBlock\">view raw</a>\n        <a href=\"https://gist.github.com/KyMidd/fb0cd10efd25909a66e7ff2016327d69#file-asdf-tf\" class=\"Link--inTextBlock\">\n          asdf.tf\n        </a>\n        hosted with &#10084; by <a class=\"Link--inTextBlock\" href=\"https://github.com\">GitHub</a>\n      </div>\n    </div>\n</div>\n",
    "stylesheet": "https://github.githubassets.com/assets/gist-embed-c38b724e3032.css"
}

The template tags are still there in the the innerHTML field, thus everything is working fine until that point.

That HTML is then rendered by substack inside a ProseMirror component, there is likely where the bug happens, since it turns the HTML into:

<link rel="stylesheet" href="https://github.githubassets.com/assets/gist-embed-c38b724e3032.css">
<div id="gist131057207" class="gist">
  <div class="gist-file" data-color-mode="light" data-light-theme="light">
    <div class="gist-data">
      <div class="js-gist-file-update-container js-task-list-container">
        <div id="file-test-txt" class="file my-2">

          <div itemprop="text" class="Box-body p-0 blob-wrapper data type-text  ">


            <div class="js-check-bidi js-blob-code-container blob-code-content">


              <div data-view-component="true" class="flash flash-warn flash-full d-flex flex-items-center">



                <span>
                  This file contains bidirectional Unicode text that may be interpreted or compiled differently than
                  what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
                  <a class="Link--inTextBlock" href="https://github.co/hiddenchars" target="_blank">Learn more about
                    bidirectional Unicode characters</a>
                </span>


                <div data-view-component="true" class="flash-action"> <a href="{{ revealButtonHref }}"
                    data-view-component="true" class="btn-sm btn"> Show hidden characters
                  </a>
                </div>
              </div>

              <span data-view-component="true" class="line-alert tooltipped tooltipped-e">



              </span>

              <table data-hpc=""
                class="highlight tab-size js-file-line-container js-code-nav-container js-tagsearch-file"
                data-tab-size="8" data-paste-markdown-skip="" data-tagsearch-lang="Text" data-tagsearch-path="test.txt">
                <tbody>
                  <tr>
                    <td id="file-test-txt-L1" class="blob-num js-line-number js-code-nav-line-number js-blob-rnum"
                      data-line-number="1"></td>
                    <td id="file-test-txt-LC1" class="blob-code blob-code-inner js-file-line">test</td>
                  </tr>
                </tbody>
              </table>
            </div>


          </div>

        </div>
      </div>

    </div>
    <div class="gist-meta">
      <a href="https://gist.github.com/Maluen/b653862672f998d55bb34a9cb67a3ec3/raw/5ad060ce56a1cf704f9bbc1ed674eccae155f64e/test.txt"
        style="float:right" class="Link--inTextBlock">view raw</a>
      <a href="https://gist.github.com/Maluen/b653862672f998d55bb34a9cb67a3ec3#file-test-txt" class="Link--inTextBlock">
        test.txt
      </a>
      hosted with ❤ by <a class="Link--inTextBlock" href="https://github.com">GitHub</a>
    </div>
  </div>
</div>

That is the template elements disappear except their content is kept, causing it to be rendered.

[Maluen]

They have a custom GitgistToDOM component that sanitizes that HTML, only keeping allowed tags and attributes.

Unfortunately template isn't in the list of allowed tags:

['address', 'article', 'aside', 'footer', 'header', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'hgroup', 'main', 'nav', 'section', 'blockquote', 'dd', 'div', 'dl', 'dt', 'figcaption', 'figure', 'hr', 'li', 'main', 'ol', 'p', 'pre', 'ul', 'a', 'abbr', 'b', 'bdi', 'bdo', 'br', 'cite', 'code', 'data', 'dfn', 'em', 'i', 'kbd', 'mark', 'q', 'rb', 'rp', 'rt', 'rtc', 'ruby', 's', 'samp', 'small', 'span', 'strong', 'sub', 'sup', 'time', 'u', 'var', 'wbr', 'caption', 'col', 'colgroup', 'table', 'tbody', 'td', 'tfoot', 'th', 'thead', 'tr']

[Maluen]

There is a way the extension can avoid the error, it's kind of ugly but it works. Since the gist HTML is fetched with a client-side HTTP call, we can patch the call itself to strip the template tags and their content from the HTML:

var _open = XMLHttpRequest.prototype.open;
window.XMLHttpRequest.prototype.open = function (method, URL) {
    var _onreadystatechange = this.onreadystatechange,
        _this = this;

    _this.onreadystatechange = function () {
        if (_this.readyState === 4 && _this.status === 200 && ~URL.indexOf('api/v1/github/gist')) {
            try {
                var data = JSON.parse(_this.responseText);

                data.innerHTML = data.innerHTML.replace(/<template ([\s\S]*?)>([\s\S]+?)<\/template>/g, ' ');

                // rewrite responseText
                Object.defineProperty(_this, 'responseText', {
                  value: JSON.stringify(data),
                  configurable: true,
                  enumerable: true,
                });
            } catch (e) {}
        }
        // call original callback
        if (_onreadystatechange) _onreadystatechange.apply(this, arguments);
    };

    // detect any onreadystatechange changing
    Object.defineProperty(this, "onreadystatechange", {
        get: function () {
            return _onreadystatechange;
        },
        set: function (value) {
            _onreadystatechange = value;
        }
    });

    return _open.apply(_this, arguments);
};

See https://stackoverflow.com/a/51594799

If we execute this code in the Substack editor on load, all subsequent gist additions will show correctly, including in the published post!

However the following code must be injected in the page itself, executing it in a content script isn't enough given it's a separate context.