Maths support (MSC2191) #2133

NickHu · 2020-09-18T15:12:11Z

This is a PR to implement maths support as described here. (TLDR: inline maths is given by HTML <span data-mx-maths="latex here">fallback representation here</span>, and display maths by div instead of span).

Fortunately, markwon already supports rendering LaTeX; it was just a
matter of enabling it, bumping its version (for inline LaTeX), and doing
some minor processing for MSC2191.

Current state:

Rendering
- Renders inline maths <span data-mx-maths=...>...</span>
- Renders display maths <div data-mx-maths=...>...</div>
- Render messages which have been edited
Sending
- $latex$ → <span data-mx-maths="latex"><code>latex</code></span>
- $latex$ → <span data-mx-maths="latex"><code>latex</code></span>
- $$latex$$ → <div data-mx-maths="latex"><code>latex</code></div>
- \[latex\] → <div data-mx-maths="latex"><code>latex</code></div>

I only implemented an inline Delimiter parser so things like

$$
latex
$$

won't be parsed correctly (requires a block parser), and apparently this class only supports having (possibly many
consecutive) single-character parsers, so \( and \[ weren't so straightforward to implement either. It seems like that with
commonmark/commonmark-java#180 inline parsing is getting an overhaul anyway, so it might be best to
wait for that to implement parsing these. It's probably fine to omit this for a first version.

A strange bug is that when rendering a message which only contains maths, it seems to draw the message with very small vertical height to the point at which it appears like a blank bit of vspace (the content is not visible). Sometimes sections of display maths are cropped at the bottom which suggests in general I'm not setting the height of the TextView properly. Any guidance on this would be appreciated.

Signed-off-by: Nick Hu <[email protected]>

Pull Request Checklist

Changes has been tested on an Android device or Android emulator with API 21
UI change has been tested on both light and dark themes
Pull request is based on the develop branch
Pull request updates CHANGES.md
Pull request includes screenshots or videos if containing UI changes
Pull request includes a sign off

tobiasBora · 2020-12-13T12:53:37Z

Can't wait for this to be merged ;-) Did you manage to solve the issue with cut texts?

a22sc · 2021-07-09T12:27:32Z

Since matrix has become more and more popular in science and universities, this feature is a must have!
I'm not deep enough into the code to review the pull request myself. Maybe @bmarty or someone else has capacities..

bmarty

Thanks for the PR. Is it possible to:

Add a lab flag to this feature, disabled by default? The impacted code is sensible, we do not want to break anything, also we do not want to downgrade the performance
Convert the Java classes you added to kotlin.

vector/build.gradle

kylrth · 2021-09-03T18:09:13Z

Where is this at? This feature is really important for me and it'd be good to get all this merged before the work becomes stale compared to develop.

antonis-tsolomitis · 2021-09-08T07:48:03Z

I agree with @kylrth This is Very important for me too as a Mathematician. I hope someone fixes this to be merged soon.

NickHu · 2021-09-08T15:12:25Z

I have ported the Java over to Kotlin and confirmed that things are working as they were before. I will try to put everything behind a labs setting next.

I still haven't managed to fix the clipping issue however. I think what happens is that JLaTeXMath schedules what it calls an AsyncDrawableScheduler over the messageView; however, this preserves the dimensions of the messageView which is (especially in display maths) smaller. I need to find some way to rerender the widget, but I'm not sure how.

NickHu · 2021-09-08T16:22:47Z

Ok, now it's gated behind Labs

cc @bmarty

NickHu · 2021-09-08T22:18:22Z

So I've determined that the 'wonkiness' with respect to clipping is a bit more subtle:

display maths gets clipped (but you can still essentially see it, just not the bottom of the integration symbol e.g.);
if a message contains only inline maths, it initially gets drawn with near zero height (and forcing a minimum height doesn't make it render correctly either); however, once you scroll until it falls out of view (dropped by the RecyclerView), and then scroll back, it renders correctly.

In any case, it's quite usable even in this state (for the impatient: you can download the APK from buildkite once it finishes building), and I don't think I have the capability to fix this. I'm sure someone more experienced with Android development would be able to figure it out pretty quickly.

All told, I think this can be merged if it's going to be behind a labs toggle. People are pretty desperate to use this now, and I'm sure once it's in there it won't be long before someone can take this the rest of the way. Remember to enable the labs setting and then restart the app; also, enable Markdown formatting if you wish to send messages with LaTeX content.

antonis-tsolomitis · 2021-09-09T06:05:29Z

for the impatient: you can download the APK from buildkite once it finishes building

I am unexperienced... can someone write how to download the apk on an android phone?

mr-c · 2021-09-09T13:01:16Z

for the impatient: you can download the APK from buildkite once it finishes building

I am unexperienced... can someone write how to download the apk on an android phone?

Go to https://buildkite.com/matrix-dot-org/element-android/builds/3299#68f085bf-2408-4a5c-8375-fbddc1c05399
Click on Artifacts and side load the relevant APK for your phone.

antonis-tsolomitis · 2021-09-09T16:13:36Z

Go to https://buildkite.com/matrix-dot-org/element-android/builds/3299#68f085bf-2408-4a5c-8375-fbddc1c05399
Click on Artifacts and side load the relevant APK for your phone.

Thanks a lot!

antonis-tsolomitis · 2021-09-09T17:53:19Z

I installed this version on a 10" android tablet. It seems that there is a size bug for the math that may be relates to the clipping problem. The attached

photo from the tablet shows chat in Greek but it is clear that the Mathematics are too large and not positioned properly. Some above the base line and some below.

antonistsolomitis · 2021-09-16T14:24:04Z

Dear @NickHu , an android phone converts the LaTeX code to math when the message is written say from element-desktop. The other way around does not work. If we write $x^2$ on the phone, the desktop app AND the phone app does not convert this to math. It remains verbatim. Can you please verify?

bmarty

Thanks a lot for the update.

I've enabled GitHub action (CI), please fix mandatory action(s) in error.
Can you add a changelog file please? See https://github.com/vector-im/element-android/blob/develop/CONTRIBUTING.md#changelog for more details

bmarty · 2021-09-17T07:52:30Z

...src/main/java/im/vector/app/features/home/room/detail/timeline/factory/MessageItemFactory.kt

@@ -484,6 +484,7 @@ class MessageItemFactory @Inject constructor(
 }
 .useBigFont(linkifiedBody.length <= MAX_NUMBER_OF_EMOJI_FOR_BIG_FONT * 2 && containsOnlyEmojis(linkifiedBody.toString()))
 .canUseTextFuture(canUseTextFuture)
+ .markwonPlugins(htmlRenderer.get().plugins)


Add this behind the lab flag too?

To be honest, I think this should not be gated behind the labs flag. I think you are supposed to use markwon this way. It just so happens that with the existing configuration, it only loads the html rendering plugin and that doesn't have any pre/post hooks so it doesn't matter if you omit this line. In any case, it shouldn't affect performance.

vector/src/main/java/im/vector/app/features/html/EventHtmlRenderer.kt

NickHu · 2021-09-18T10:31:21Z

Dear @NickHu , an android phone converts the LaTeX code to math when the message is written say from element-desktop. The other way around does not work. If we write $x^2$ on the phone, the desktop app AND the phone app does not convert this to math. It remains verbatim. Can you please verify?

Please confirm you have both

enabled Markdown input in preferences
enabled the Maths Labs feature and restarted the app (clear it from recent apps and launch it again)

antonis-tsolomitis · 2021-09-18T15:21:24Z

Ah, yes I missed the Markdown setting. So now it works for inline math on a phone. There are two problems. One is that displaymath does not work (or I miss one more setting (?)):

Remaining issues include proper vertical spacing (except clipping) as seen below:

(matrix-org/matrix-spec-proposals#2191) Firstly, this implements a commonmark-java plugin which is solely used to parse LaTeX input in the composer box, so that they can be rendered into `<span data-mx-maths=...>fallback</span>` and `<div data-mx-maths=...>fallback</div>` for inline and display maths respectively in the sent message. Secondly, received messages of this form are pre-processed by a simple regex into a form which markwon (which performs the rendering) expects.

NickHu · 2021-09-24T12:52:18Z

I rebased onto develop, but I had to downgrade the markwon version relative to there (it was updated just over a week ago) because of noties/Markwon#264 (closed but never fixed). It's still a version of markwon newer than what the current stable version uses.

cc @bmarty

Esokrates · 2021-10-04T10:15:04Z

@NickHu Tested this build here: https://buildkite.com/matrix-dot-org/element-android/builds/3581#e9486708-9309-47d5-9810-665fe4d8ab68

Displayed math is not rendered at all here, no matter if its $$ or \[.

bmarty · 2021-10-13T09:36:32Z

@NickHu Tested this build here: https://buildkite.com/matrix-dot-org/element-android/builds/3581#e9486708-9309-47d5-9810-665fe4d8ab68

Displayed math is not rendered at all here, no matter if its $$ or \[.

Same here. Markdown is enabled, new lab feature is enabled, but latex is not rendered, if I send for instance $$x^2$$, it renders x^2 in the timeline

The source of the Event is

Also I'm wondering why the html tag is div and not span as in the PR description.

bmarty · 2021-10-13T09:39:07Z

dependencies.gradle

@@ -14,7 +14,7 @@ def kotlinCoroutines = "1.5.1"
 def dagger = "2.38.1"
 def retrofit = "2.9.0"
 def arrow = "0.8.2"
-def markwon = "4.6.2"
+def markwon = "4.3.1"


Dependabot will upgrade this dependency, as soon as the PR is merged. Can you add a comment just above to explain why we should stick to this version? A related issue we can track to check if we can safely upgrade would be nice too, or decribe a test that we should do to check that upgrading the lib does not break Math Latex rendering

NickHu · 2021-10-13T12:10:59Z

@NickHu Tested this build here: https://buildkite.com/matrix-dot-org/element-android/builds/3581#e9486708-9309-47d5-9810-665fe4d8ab68
Displayed math is not rendered at all here, no matter if its $$ or \[.

Same here. Markdown is enabled, new lab feature is enabled, but latex is not rendered, if I send for instance $$x^2$$, it renders x^2 in the timeline

The source of the Event is

I just tested it and it's working for me. Did you remember to fully restart the app after enabling the labs setting, and enable Markdown formatting?

@antonis-tsolomitis I believe that if you remove the spaces, i.e. $$x^2$$ rather than $$ x^2 $$ it should work.

Also I'm wondering why the html tag is div and not span as in the PR description.

The MSC defines div for display maths, and span for inline.

antonistsolomitis · 2021-10-13T14:55:44Z

Yes I just checked and it worked. It seems there is another bug though, hopefully easy to solve. It does not like > (maybe < too) either display or inline. If I write \[\mathcal{H}^s (A)=\sup_{\delta > 0}\] it appears fine on desktop but fails on android. It shows the source as \[\mathcal{H}^s (A)=\sup_{\delta > 0}\] Is it possible it first changes > to > and then runs katex and of course fails?
If I write \geq instead of > it works. If I write \greater (from unicode-math) it does not know \greater.

bmarty · 2021-10-14T12:06:41Z

vector/src/main/res/xml/vector_settings_labs.xml

+ <im.vector.app.core.preference.VectorSwitchPreference
+ android:defaultValue="false"
+ android:key="SETTINGS_LABS_ENABLE_LATEX_MATHS"
+ android:title="@string/labs_enable_latex_maths"/>


You should mention in the summary that user has to restart the app for the change to be taken into account.

bmarty · 2021-12-31T15:44:52Z

I will handle the remaining work on my branch.

antonis-tsolomitis · 2022-01-11T07:18:11Z

Is there any estimate when this will land in Playstore?

NickHu marked this pull request as ready for review September 25, 2020 09:30

NickHu force-pushed the maths branch from a5861e2 to de4a670 Compare October 19, 2020 16:09

NickHu mentioned this pull request Nov 4, 2020

LaTeX rendering in element-web using KaTeX matrix-org/matrix-react-sdk#5244

Merged

NickHu mentioned this pull request Dec 7, 2020

LaTeX rendering matrix-org/matrix-android-sdk#566

Open

NickHu force-pushed the maths branch from de4a670 to 12c3936 Compare May 19, 2021 15:26

bmarty requested changes Jul 27, 2021

View reviewed changes

vector/build.gradle Outdated Show resolved Hide resolved

NickHu force-pushed the maths branch from 12c3936 to bcc1e84 Compare September 8, 2021 15:06

NickHu force-pushed the maths branch from bcc1e84 to 35754b1 Compare September 8, 2021 16:22

NickHu requested a review from bmarty September 8, 2021 16:22

NickHu mentioned this pull request Sep 9, 2021

Gitter feature parity overview issue element-hq/roadmap#26

Open

48 tasks

MadLittleMods added A-Timeline Z-Community-PR Issue is solved by a community member's PR labels Sep 17, 2021

bmarty requested changes Sep 17, 2021

View reviewed changes

NickHu force-pushed the maths branch from 0847ff1 to 6255f6e Compare September 24, 2021 11:43

NickHu requested a review from bmarty September 24, 2021 12:48

Downgrade markwon version and enable LaTeX rendering for messages

78b870c

NickHu added 5 commits September 24, 2021 13:50

Migrate commonmark extension to kotlin

4c45a69

Gate LaTeX maths behind labs flag

2262cd4

Use PrecomputedFutureTextSetterCompat

cf1a938

Fix lints and add changelog.d entry

0a498be

NickHu force-pushed the maths branch from 6255f6e to 0a498be Compare September 24, 2021 12:51

SimonBrandner mentioned this pull request Oct 13, 2021

Maths support matrix-org/matrix-react-sdk#3251

Closed

bmarty reviewed Oct 13, 2021

View reviewed changes

bmarty reviewed Oct 14, 2021

View reviewed changes

bmarty added this to Inbox in Element Android PRs lifecycle via automation Dec 31, 2021

bmarty moved this from Inbox to Need update from submitter in Element Android PRs lifecycle Dec 31, 2021

bmarty moved this from Need update from submitter to To be finalized by us in Element Android PRs lifecycle Dec 31, 2021

bmarty self-assigned this Dec 31, 2021

Merge branch 'develop' into maths

95b1ae9

bmarty changed the base branch from develop to feature/bma/math_final December 31, 2021 15:41

bmarty merged commit 4a1c924 into element-hq:feature/bma/math_final Dec 31, 2021

Element Android PRs lifecycle automation moved this from To be finalized by us to Done Dec 31, 2021

This was referenced Jan 3, 2022

Safe epoxy char sequence #4837

Merged

Support LaTex Math #4838

Merged

NickHu deleted the maths branch January 11, 2022 10:56

anoadragon453 mentioned this pull request May 13, 2022

MSC2191: Markup for mathematical messages matrix-org/matrix-spec-proposals#2191

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Maths support (MSC2191) #2133

Maths support (MSC2191) #2133

NickHu commented Sep 18, 2020 •

edited

Loading

tobiasBora commented Dec 13, 2020

a22sc commented Jul 9, 2021

bmarty left a comment •

edited

Loading

kylrth commented Sep 3, 2021

antonis-tsolomitis commented Sep 8, 2021

NickHu commented Sep 8, 2021

NickHu commented Sep 8, 2021

NickHu commented Sep 8, 2021 •

edited

Loading

antonis-tsolomitis commented Sep 9, 2021

mr-c commented Sep 9, 2021

antonis-tsolomitis commented Sep 9, 2021

antonis-tsolomitis commented Sep 9, 2021 •

edited

Loading

antonistsolomitis commented Sep 16, 2021

bmarty left a comment •

edited

Loading

bmarty Sep 17, 2021

NickHu Sep 24, 2021

NickHu commented Sep 18, 2021

antonis-tsolomitis commented Sep 18, 2021

NickHu commented Sep 24, 2021 •

edited

Loading

Esokrates commented Oct 4, 2021 •

edited

Loading

bmarty commented Oct 13, 2021

bmarty Oct 13, 2021

NickHu commented Oct 13, 2021

antonistsolomitis commented Oct 13, 2021 •

edited

Loading

bmarty Oct 14, 2021

bmarty commented Dec 31, 2021

antonis-tsolomitis commented Jan 11, 2022

Maths support (MSC2191) #2133

Maths support (MSC2191) #2133

Conversation

NickHu commented Sep 18, 2020 • edited Loading

Pull Request Checklist

tobiasBora commented Dec 13, 2020

a22sc commented Jul 9, 2021

bmarty left a comment • edited Loading

Choose a reason for hiding this comment

kylrth commented Sep 3, 2021

antonis-tsolomitis commented Sep 8, 2021

NickHu commented Sep 8, 2021

NickHu commented Sep 8, 2021

NickHu commented Sep 8, 2021 • edited Loading

antonis-tsolomitis commented Sep 9, 2021

mr-c commented Sep 9, 2021

antonis-tsolomitis commented Sep 9, 2021

antonis-tsolomitis commented Sep 9, 2021 • edited Loading

antonistsolomitis commented Sep 16, 2021

bmarty left a comment • edited Loading

Choose a reason for hiding this comment

bmarty Sep 17, 2021

Choose a reason for hiding this comment

NickHu Sep 24, 2021

Choose a reason for hiding this comment

NickHu commented Sep 18, 2021

antonis-tsolomitis commented Sep 18, 2021

NickHu commented Sep 24, 2021 • edited Loading

Esokrates commented Oct 4, 2021 • edited Loading

bmarty commented Oct 13, 2021

bmarty Oct 13, 2021

Choose a reason for hiding this comment

NickHu commented Oct 13, 2021

antonistsolomitis commented Oct 13, 2021 • edited Loading

bmarty Oct 14, 2021

Choose a reason for hiding this comment

bmarty commented Dec 31, 2021

antonis-tsolomitis commented Jan 11, 2022

NickHu commented Sep 18, 2020 •

edited

Loading

bmarty left a comment •

edited

Loading

NickHu commented Sep 8, 2021 •

edited

Loading

antonis-tsolomitis commented Sep 9, 2021 •

edited

Loading

bmarty left a comment •

edited

Loading

NickHu commented Sep 24, 2021 •

edited

Loading

Esokrates commented Oct 4, 2021 •

edited

Loading

antonistsolomitis commented Oct 13, 2021 •

edited

Loading