Desktop: Resolves #5224: Add Yaml exporter/importer #5465

CalebJohn · 2021-09-15T06:33:56Z

I realized that this is going to be decently sized PR and it's taking me awhile so I should get it in front of people sooner rather than later.

This implements the spec described in #5224. Currently I've only added tests for note import, but I have tested note export quite a bit. Another thing missing from this PR is proper handling of HTML files, they will need special handling so that they remain valid HTML.

If there are any suggestions about the feature in general, now would be a good time to start another dialogue.

laurent22 · 2021-09-15T09:25:17Z

Sorry I've pressed a button earlier, which I thought was to run CI, but it was actually to remove the draft status. Unfortunately it looks it's not possible to restore that status (always found it weird that GitHub doesn't allow that).

laurent22

Thanks for the pull request @CalebJohn. I didn't notice any major issue, just minor things I've mentioned in comments.

laurent22 · 2021-09-15T09:33:27Z

packages/lib/services/interop/InteropService_Exporter_Yaml.ts

+			return fieldOrder.indexOf(a) - fieldOrder.indexOf(b);
+		};
+
+		return yaml.dump(md, { sortKeys: sort });


I wonder how the library handles fields that contain new lines? I think the note title in particular could contain some.

packages/lib/services/interop/InteropService_Importer_Md.ts

packages/lib/services/interop/InteropService_Importer_Yaml.test.ts

packages/lib/services/interop/InteropService_Importer_Yaml.ts

packages/lib/time.ts

laurent22 · 2021-09-15T09:49:59Z

Also a failing test: https://github.com/laurent22/joplin/pull/5465/checks?check_run_id=3606964088#step:8:494

A time zone issue?

CalebJohn · 2021-09-16T07:27:08Z

A time zone issue?

Look's like it's an error of 7 hours, and given that I'm in the UTC-7 timezone, I'd say it's a time zone issue...

I've changed the way that check works so the tests should pass now.

laurent22 · 2021-09-18T09:47:38Z

Thanks for the update. I can't think of any remaining issue for now, so when it's ready to review please let me know.

This new system inherits from the MD exporter/importer but uses a yaml metadata block at the start of a note to store useful metadata

await while adding tags Time -> time make importedNotes private again

Sort exported tags Adjust boolean field to comply with FAILSAFE yaml

Fix arbitrary date format import Add extra tests

CalebJohn · 2021-09-27T07:47:39Z

@laurent22 This is ready for another review!

laurent22

Thanks @CalebJohn, I think it's nearly there. Please could you also add a spec under readme/spec? I guess just pasting there an example of file would be sufficient with perhaps an explanation of what this format is.

laurent22 · 2021-09-28T14:23:08Z

packages/lib/services/interop/InteropService.ts

@@ -58,6 +58,14 @@ export default class InteropService {
 					isNoteArchive: false, // Tells whether the file can contain multiple notes (eg. Enex or Jex format)
 					description: _('Markdown'),
 				},
+				{
+					...defaultImportExportModule(ModuleType.Importer),
+					format: 'yaml',


Shouldn't that be "md" too? Also how does it work with the CLI tool - is it possible to disambiguate and select one format or the other (regular MD or MD+YAML)? I remember having the same issue with "ENEX as MD" vs "ENEX as HTML".

My understanding is that this format is just a short name for the module. So for the CLI tool, you could use --format yaml. Although based on your other comment this will need to be changed, I'm thinking md-frontmatter.
How did you solve the enex issue? Based on the cli docs, it looks like it still exists.

Right, I got confused. For ENEX, what I had done is add an "output" parameter to the import command, so you can import an ENEX file and set the output to create notes in either MD or HTML format.

But I guess that wouldn't apply in your case since you always create Markdown notes? But actually that makes me wonder - you've added html as supported format. In that case, how do you handle it? I'm not seeing any code that would process html files?

And for the format in general, you're right, it just need to be a unique identifier. "md-frontmatter" would indeed make sense.

I ended up going with md_frontmatter because it works better with JavaScript.

The HTML files are html in file extension only (this is the same for the Md export). It's a way of indicating to the user (and any system that ingests the export) that the file contains html and not markdown. But i still wanted the file to be renderable by a markdown editor. So the output is all technically markdown, but html files contain HTML markup.

It could also make sense to hook this in to the the html exporter/importer instead. But updating that exporter/importer would make this already large PR a bit too big for my taste.

packages/lib/services/interop/InteropService.ts

Remove whitespace stripping The original reason for this is that non-technical users aren't used to whitespace sensitivity so it made sense to ignore whitespace. This, however, breaks multiline yaml strings. I realized that the original reasoning is flawed, this feature needs to be accessible to non-technical users, but editing/creating the yaml will still be in the domain of technical users.

Fix error with time being return in utc rather than local

removed category field from being used as tags (it's more akin to folder) Added lastmod for hugo handling https://jekyllrb.com/docs/front-matter/ https://gohugo.io/content-management/front-matter

This case happens when geo location is turned off. So there's no information to export

This ensures greater overlap with external tools (e.g. pandoc)

This ensures datetimes are specified in UTC

This is a universal change so that only the part in front of the _ will be displayed in the GUI, the assumption being that the description will differentiate the formats

laurent22

Thanks for the update @CalebJohn. Would it be possible to create a spec document that describes:

The list of exported fields
Maybe the supported imported fields (doesn't need to be as detailed)
How dates are handled when imported. For example, what if the date has no time zone, or a time zone, or UTC
How dates are handled when exported (I guess all UTC now?)
An example of valid Markdown + Front Matter document

laurent22 · 2021-10-07T11:27:24Z

packages/app-cli/tests/support/test_notes/yaml/full.md

+title: Test Note Title
+updated: 2019-05-01 16:54
+created: 2019-05-01 16:54
+Source: https://joplinapp.org
+author: Joplin
+latitude: 37.084021
+longitude: -94.51350100
+altitude: 0.0000
+Completed?: false
+due: 2021-08-22 00:00
+tags:
+  - Joplin
+  - nOte
+  - pencil
+---


There's a mix of lowercase (author) and uppercase (Completed) in there. Is it on purpose? Also the dates don't seem to be UTC?

Both of those are on purpose, the importer is meant to be quite flexible and handles poor formatting (case) and as many date formats as possible. If no timezone is provided, the importer will interpret the timezone as local. There's another test that checks notes with ISO 8601 time zones.

That makes sense, thanks for clarifying.

Correct some capitalization issues in the Exporter

CalebJohn · 2021-10-07T20:27:28Z

@laurent22 I've added a spec document now, let me know if you want anything else added to it.

laurent22

Thanks for the update.

I've mostly left comments about setting types for variables. There was a few bugs no long ago that could have been avoided if types had been set explicitly so that's why I'm trying to make sure that everything has a type (unless setting one is needlessly complicated).

Quite often it's ok to define the interface in the class file or even in the function itself. It doesn't really matter how it's done - as long as it's there TypeScript will pick it up.

packages/lib/services/interop/InteropService_Exporter_Md_frontmatter.ts

laurent22 · 2021-10-08T10:16:19Z

packages/app-cli/tests/support/test_notes/yaml/full.md

+title: Test Note Title
+updated: 2019-05-01 16:54
+created: 2019-05-01 16:54
+Source: https://joplinapp.org
+author: Joplin
+latitude: 37.084021
+longitude: -94.51350100
+altitude: 0.0000
+Completed?: false
+due: 2021-08-22 00:00
+tags:
+  - Joplin
+  - nOte
+  - pencil
+---


That makes sense, thanks for clarifying.

packages/lib/services/interop/InteropService_Exporter_Md.ts

packages/lib/services/interop/InteropService_Exporter_Md_frontmatter.ts

packages/lib/services/interop/InteropService_Importer_Md_frontmatter.ts

readme/spec/interop_with_frontmatter.md

laurent22 · 2021-10-16T08:59:13Z

I think it's all good now, thanks a lot for adding this @CalebJohn!

…porter/importer (laurent22#5465)

commit 5cd4537 Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 17:29:13 2021 +0000 Server: Rename "ReadOnly" mode to "ReadAndClear" to avoid any confusion commit ec8b9b8 Author: Kenichi Kobayashi <[email protected]> Date: Fri Nov 12 02:28:32 2021 +0900 Desktop: Fixes laurent22#5694: Sync-scroll support for HTML notes (laurent22#5695) commit 4bfb4db Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 16:36:43 2021 +0000 Server: Fixed S3 connection string record in db commit 4fc3bcb Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 16:32:34 2021 +0000 Server: Fixed S3 storage connection and improved connectiob checks commit 01826d9 Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 16:31:37 2021 +0000 ignore files commit 73137cf Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 16:12:46 2021 +0000 Server: Hide credentials from the log commit b58ea0f Author: Helmut K. C. Tessarek <[email protected]> Date: Thu Nov 11 10:50:15 2021 -0500 Desktop: Add shortcut for bulleted list (laurent22#5698) Ref: https://discourse.joplinapp.org/t/shortcut-for-lists/21646 commit d7e0877 Author: Kenichi Kobayashi <[email protected]> Date: Fri Nov 12 00:43:32 2021 +0900 Desktop: Resolves laurent22#4827, Resolves laurent22#5652: Fixed and improve laggy scroll in text editor (laurent22#5606) commit f495db1 Author: Kenichi Kobayashi <[email protected]> Date: Fri Nov 12 00:33:37 2021 +0900 Desktop: Sort Order Buttons and Per-Notebook Sort Order (laurent22#5437) commit e4d5e9c Author: Kenichi Kobayashi <[email protected]> Date: Fri Nov 12 00:31:20 2021 +0900 Desktop: Fixes laurent22#5582: Currently opened note is not updated after sync (5582) (laurent22#5711) commit 72f336a Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 15:24:21 2021 +0000 Server v2.6.8 commit dfa5f8b Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 15:21:27 2021 +0000 Server: Fixed S3 configuration commit 3cdbd6d Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 14:44:06 2021 +0000 Server v2.6.7 commit c6dec0a Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 14:40:23 2021 +0000 Server: Added command to test a storage connection commit 4879edc Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 14:06:27 2021 +0000 Server v2.6.6 commit 005f720 Author: Laurent Cozic <[email protected]> Date: Thu Nov 11 13:59:05 2021 +0000 Server: Added command to migrate content to different storage commit 725c79d Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 15:40:32 2021 +0000 Desktop: Fixed button to upgrade a master key commit 0de6e9e Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 14:47:26 2021 +0000 All: Fixed issue that could cause application to needlessly lock the sync target commit 7c3785e Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 12:51:41 2021 +0000 Doc: Update privacy policy commit 5c2a0ed Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 12:20:43 2021 +0000 Doc: Update privacy policy commit fc64c82 Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 12:08:10 2021 +0000 Server v2.6.5 commit 08f420c Merge: d76646a cc23a8b Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 12:07:32 2021 +0000 Merge branch 'dev' into release-2.6 commit cc23a8b Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 16:12:29 2021 +0000 Server v2.6.4 commit 9ffa29f Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 12:03:45 2021 +0000 Chore: Add tests for storage driver loading commit 7431da9 Author: Laurent Cozic <[email protected]> Date: Wed Nov 10 11:48:06 2021 +0000 Server: Lazy-load storage drivers commit 4deeed0 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 18:33:28 2021 +0000 Desktop, Mobile: Fixes laurent22#5687: Fixed issue with parts of HTML notes not being displayed in some cases commit f8d9601 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 17:49:05 2021 +0000 Tools: Restored HTML to ENEX tests that had been deleted in a refactoring commit 89179c2 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 16:42:50 2021 +0000 Desktop, Cli: Add support for more style of highlighted texts when importing ENEX files commit d76646a Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 16:12:29 2021 +0000 Server v2.6.4 commit 485c0d0 Author: Laurent <[email protected]> Date: Tue Nov 9 16:05:42 2021 +0000 Server: Allow storing item content in database, filesystem or S3 (depending on config) (laurent22#5602) commit 6b31609 Author: agerardin <[email protected]> Date: Tue Nov 9 10:50:50 2021 -0500 Plugins: Allow posting messages from plugin to webview (laurent22#5569) commit 200ba85 Author: Kenichi Kobayashi <[email protected]> Date: Wed Nov 10 00:47:25 2021 +0900 Deskop: Fixes flickering when switching notes while the Note Search Bar is visible (laurent22#5548) commit 7b3ad32 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 15:28:38 2021 +0000 Update translations commit 3745cd7 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 15:26:45 2021 +0000 Tools: Do not process context when running build-translation tool commit 920f2d9 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 13:49:20 2021 +0000 Revert "Update translations" This reverts commit f800ca0. Reverting for now due to some translations being incorrectly marked as fuzzy. commit f800ca0 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 13:16:09 2021 +0000 Update translations commit 33be306 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 13:14:41 2021 +0000 Tools: Fixed missing translations when running build-translation tool commit 3782255 Author: Laurent Cozic <[email protected]> Date: Tue Nov 9 12:20:07 2021 +0000 Tools: Fixed build-translation script to handle .ts and .tsx files directly commit 0ab2352 Author: Laurent <[email protected]> Date: Mon Nov 8 17:10:33 2021 +0000 Tools: Detect missing translation strings on CI (laurent22#5688) commit 7525661 Author: Laurent Cozic <[email protected]> Date: Mon Nov 8 15:38:44 2021 +0000 Security: Ensure Markdown links that contain single quotes are correctly escaped commit b328094 Author: Helmut K. C. Tessarek <[email protected]> Date: Mon Nov 8 10:30:18 2021 -0500 update en_US.po commit 4f0f1af Author: Helmut K. C. Tessarek <[email protected]> Date: Mon Nov 8 10:22:08 2021 -0500 Update translations commit 021ce14 Author: Laurent Cozic <[email protected]> Date: Mon Nov 8 15:21:20 2021 +0000 Server v2.6.3 commit c4017e5 Author: Laurent Cozic <[email protected]> Date: Mon Nov 8 10:00:11 2021 +0000 Fixed a few strings commit ec2c174 Author: Laurent Cozic <[email protected]> Date: Sun Nov 7 17:36:48 2021 +0000 Tools: Fixed package version numbers commit 3e5ad0a Author: Laurent Cozic <[email protected]> Date: Sun Nov 7 16:41:39 2021 +0000 Desktop, Cli: Fixes laurent22#5653: Long resource filenames were being incorrectly cut commit 05e390d Author: Laurent Cozic <[email protected]> Date: Sun Nov 7 15:41:04 2021 +0000 Tools: Fixed build-translation script commit 31ce0f4 Author: Laurent Cozic <[email protected]> Date: Sun Nov 7 11:55:34 2021 +0000 Tools: Show more info in IOS release script commit d19551b Author: Laurent Cozic <[email protected]> Date: Sun Nov 7 11:55:11 2021 +0000 Chore: Set correct iOS version ID commit dacd697 Author: Laurent Cozic <[email protected]> Date: Sun Nov 7 11:53:39 2021 +0000 iOS: Fixes laurent22#5671: Fixed iOS 12 crash that prevents the app from starting commit 70d5c7a Author: Laurent Cozic <[email protected]> Date: Fri Nov 5 12:17:05 2021 +0000 Server: Set resource content size when viewing published note commit 42caab6 Author: Daniel Bretoi <[email protected]> Date: Sat Nov 6 21:32:18 2021 -0700 Doc: CLI: update to include local filesystem sync (laurent22#5684) * terminal doc update to include local filesystem sync * fix typo commit 01b63ad Author: NiceYoyo <[email protected]> Date: Sun Nov 7 05:30:04 2021 +0100 All: Translation: Update de_DE.po (laurent22#5682) * Update de_DE.po * Update de_DE.po commit ae4013d Author: felipeviggiano <[email protected]> Date: Sun Nov 7 01:29:38 2021 -0300 All: Translation: Update pt_BR.po (laurent22#5680) * Updates on pt_BR.po file * Updates on pt_BR.po file commit 298e85f Author: Laurent Cozic <[email protected]> Date: Thu Nov 4 16:43:34 2021 +0000 iOS: Set min supported iOS version to 13.0 commit 9e1cb9d Author: Laurent Cozic <[email protected]> Date: Thu Nov 4 12:49:51 2021 +0000 Server: Immediately ask user to set password after Stripe checkout commit 373c041 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 17:39:30 2021 +0000 Doc: Fixed H3 header size commit af19865 Author: Laurent <[email protected]> Date: Wed Nov 3 16:24:40 2021 +0000 All, Server: Add support for sharing notes when E2EE is enabled (laurent22#5529) commit a0d2304 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 15:20:17 2021 +0000 Migration can be null commit 7ad73df Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 15:18:20 2021 +0000 Server: Display latest migration name after auto-migration commit ce5c5d6 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 14:00:16 2021 +0000 Server: Disable mailer service if no-reply email is not set commit 8c6d78e Merge: a65c424 e6d3396 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 12:55:28 2021 +0000 Merge branch 'dev' into release-2.6 commit a65c424 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 12:55:01 2021 +0000 Server v2.6.2 commit e6d3396 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 12:54:08 2021 +0000 lock files commit 190550f Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 12:53:25 2021 +0000 Tools: Fixed server tests commit 030b18d Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 12:27:39 2021 +0000 Desktop release v2.6.1 commit f1bfcfd Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 12:27:28 2021 +0000 Server v2.6.1 commit 47a31c4 Author: Laurent <[email protected]> Date: Wed Nov 3 12:26:26 2021 +0000 All, Server: Add support for faster built-in sync locks (laurent22#5662) commit 630a400 Author: Kenichi Kobayashi <[email protected]> Date: Wed Nov 3 21:10:46 2021 +0900 Desktop: Resolves laurent22#2242: Implements Sync-Scroll for Markdown Editor and Viewer (laurent22#5512) commit 725abbc Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 11:44:43 2021 +0000 Update website commit 84ec845 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 11:34:14 2021 +0000 CLI v2.6.1 commit 5892a06 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 11:18:03 2021 +0000 Releasing sub-packages commit cb26ab4 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 11:15:51 2021 +0000 Tools: Trying to fix Lerna publishing with automation token commit 74fcd47 Author: Laurent Cozic <[email protected]> Date: Wed Nov 3 11:10:06 2021 +0000 Releasing sub-packages commit c7406f3 Author: Laurent Cozic <[email protected]> Date: Tue Nov 2 20:56:26 2021 +0000 Android 2.6.1 commit d22a29e Author: Laurent Cozic <[email protected]> Date: Tue Nov 2 15:25:42 2021 +0000 ios-v12.5.1 commit bcd568a Author: Laurent Cozic <[email protected]> Date: Tue Nov 2 20:07:13 2021 +0000 Setup new release 2.6 commit 66e79cc Author: Laurent Cozic <[email protected]> Date: Tue Nov 2 16:33:53 2021 +0000 Mobile: Upgraded React Native from 0.64 to 0.66 commit 08ee2b2 Merge: b5d792c 57b8aa1 Author: Laurent Cozic <[email protected]> Date: Tue Nov 2 14:47:19 2021 +0000 Merge branch 'release-2.5' into dev commit 57b8aa1 Author: Laurent Cozic <[email protected]> Date: Tue Nov 2 14:46:52 2021 +0000 Server v2.5.10 commit b5d792c Author: Laurent Cozic <[email protected]> Date: Tue Nov 2 12:51:59 2021 +0000 Server: Improved env variable handling to make it self documenting and enforce type checking commit 3704413 Author: Laurent Cozic <[email protected]> Date: Mon Nov 1 19:20:36 2021 +0000 Server: Improved logging and rendering of low level middleware errors commit c491741 Author: Laurent Cozic <[email protected]> Date: Mon Nov 1 07:57:10 2021 +0000 Doc: Added forum assets commit b80242d Author: Laurent Cozic <[email protected]> Date: Mon Nov 1 07:41:36 2021 +0000 Desktop release v2.5.10 commit 59ea52f Merge: 3813b10 1eda835 Author: Laurent Cozic <[email protected]> Date: Mon Nov 1 07:41:18 2021 +0000 Merge branch 'dev' into release-2.5 commit 1eda835 Author: Laurent Cozic <[email protected]> Date: Mon Nov 1 00:52:32 2021 +0000 Desktop release v2.5.9 commit 6012783 Author: Laurent Cozic <[email protected]> Date: Mon Nov 1 07:38:06 2021 +0000 Desktop: Fixed crash on certain Linux distributions when importing or exporting a file Ref: https://discourse.joplinapp.org/t/20702/37 commit 3813b10 Author: Laurent Cozic <[email protected]> Date: Mon Nov 1 00:52:32 2021 +0000 Desktop release v2.5.9 commit c5569ef Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 18:33:06 2021 +0000 All: Fixed potential infinite loop when Joplin Server session is invalid commit 401f1b1 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 18:31:40 2021 +0000 Tools: Fixed tests when using Joplin Server as sync target commit 99ea4b7 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 18:29:47 2021 +0000 Server: Fixed issue that could cause server to return empty items in some rare cases commit 22d472d Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 17:36:54 2021 +0000 Tools: Added testing command to create many items in parallel commit ab86812 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 11:39:12 2021 +0000 Update website commit b678d20 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 11:25:07 2021 +0000 Android 2.5.5 commit 1f96eb7 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 10:38:11 2021 +0000 Lock files commit e453cd3 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 10:35:26 2021 +0000 Tools: Build app before releasing Android app commit 232128c Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 10:29:15 2021 +0000 Android 2.5.4 commit dbdb82d Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 10:05:33 2021 +0000 Desktop release v2.5.8 commit 14ddf40 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 10:03:37 2021 +0000 Fixed French translation commit 4128be9 Author: Laurent Cozic <[email protected]> Date: Sun Oct 31 09:32:08 2021 +0000 Mobile: Capitalise first word of sentence in beta editor commit 8de9032 Author: Brett Bender <[email protected]> Date: Sat Oct 30 10:00:01 2021 -0700 Deskop: Fixed shortcut to focus the note body (laurent22#5597) commit 0b01b5b Author: Laurent Cozic <[email protected]> Date: Sat Oct 30 17:51:19 2021 +0100 Desktop, Mobile: Resolves laurent22#5593: Enable safe mode for Markdown editor too commit aa3cbbd Author: Laurent Cozic <[email protected]> Date: Sat Oct 30 17:47:16 2021 +0100 Desktop, Mobile: Fixes laurent22#5593: Do not render very large code blocks to prevent app from freezing commit 2d1a6da Author: Laurent Cozic <[email protected]> Date: Sat Oct 30 15:20:26 2021 +0100 Revert "Desktop: Upgrade to Electron 15.1.3" No longer necessary because it was to try to fix this bug, but it doesn't fix it: https://discourse.joplinapp.org/t/20702/32 This reverts commit 9704b29. commit 365e152 Author: Laurent Cozic <[email protected]> Date: Sat Oct 30 11:51:40 2021 +0100 Mobile: Add padding around beta text editor commit 6393996 Author: Andreas Deininger <[email protected]> Date: Sat Oct 30 11:45:02 2021 +0200 Doc: Fixing typos (laurent22#5644) commit f7a18ba Author: Laurent Cozic <[email protected]> Date: Sat Oct 30 10:37:56 2021 +0100 Server: Add unique constraint on name and owner ID of items table In theory, the server enforces uniquness because when doing a PUT operation it either creates the items if it doesn't exist, or overwrite it. However, there's race condition that makes it possible for multiple items with the same name being created per user. So we add this constraint to ensure that any additional query would fail (which can be recovered by repeating the request). commit bb06e56 Author: Laurent Cozic <[email protected]> Date: Fri Oct 29 18:47:11 2021 +0100 Added better-sqlite database driver, although we cannot use it for now commit 4e1f315 Author: Laurent Cozic <[email protected]> Date: Fri Oct 29 12:12:30 2021 +0100 Desktop release v2.5.7 commit 9704b29 Author: Laurent Cozic <[email protected]> Date: Fri Oct 29 12:11:25 2021 +0100 Desktop: Upgrade to Electron 15.1.3 commit 2133193 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 23:09:16 2021 +0100 Android 2.5.3 commit 7d70dea Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 22:27:47 2021 +0100 Desktop release v2.5.6 commit e247be1 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 22:25:27 2021 +0100 Desktop: Fixed default migration logic commit 9704e75 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 21:23:40 2021 +0100 Android 2.5.2 commit e4403d4 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 20:56:40 2021 +0100 Desktop release v2.5.5 commit 251400c Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 20:44:25 2021 +0100 Server v2.5.9 commit 6c6e2a6 Merge: 3359ea3 4a2af32 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 20:43:14 2021 +0100 Merge branch 'dev' into release-2.5 commit 4a2af32 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 20:41:46 2021 +0100 Server: Remove session expiration for now commit 9d88324 Author: milotype <[email protected]> Date: Thu Oct 28 18:23:52 2021 +0200 All: Translation: Update hr_HR.po (laurent22#5640) … this time with Poedit ;-) commit e0a12c7 Author: Daeraxa <[email protected]> Date: Thu Oct 28 17:20:44 2021 +0100 Doc: Add section about restoreNoteRevision command to FAQ for note restoration (laurent22#5638) * docs: add section about restoreNoteRevision command to restore accidentally deleted notes * docs:add link to wiki post commit 3359ea3 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 17:07:44 2021 +0100 Server v2.5.8 commit 2cbcfa5 Author: Laurent Cozic <[email protected]> Date: Thu Oct 28 17:06:16 2021 +0100 Slow down migration commit 0a7fdac Author: Laurent Cozic <[email protected]> Date: Wed Oct 27 19:40:32 2021 +0100 Server v2.5.7 commit a753429 Author: Laurent Cozic <[email protected]> Date: Wed Oct 27 19:38:45 2021 +0100 Server: Fixed items.owner_id migration commit dca13b3 Author: Laurent Cozic <[email protected]> Date: Wed Oct 27 19:29:54 2021 +0100 Server: Moved CLI commands to separate files commit 5c1cef8 Author: Nicholas Axel <[email protected]> Date: Thu Oct 28 01:15:38 2021 +0700 All: Translation: Update id_ID.po (laurent22#5625) * Updated Indonesian Fixed some translations Added some missing translations * Add files via upload * Add files via upload * Add files via upload commit 9ba90b5 Author: Laurent Cozic <[email protected]> Date: Wed Oct 27 16:39:45 2021 +0100 Server: Fixed Stripe portal page redirection commit f1c4d35 Author: Laurent Cozic <[email protected]> Date: Wed Oct 27 16:31:19 2021 +0100 Server: Fixed owner_id migration for SQLite commit b0e3e1b Author: Laurent Cozic <[email protected]> Date: Wed Oct 27 16:21:21 2021 +0100 Server v2.5.6 commit b655f27 Author: Laurent Cozic <[email protected]> Date: Wed Oct 27 16:18:42 2021 +0100 Server: Added item owner ID, and allow disabling db auto-migrations commit 0ada1df Author: Laurent Cozic <[email protected]> Date: Tue Oct 26 12:34:22 2021 +0100 Server: Expire sessions after 12 hours commit 134bc91 Author: theCollectiv <[email protected]> Date: Tue Oct 26 13:01:12 2021 +0200 Doc: Mention that drag and drop can be used to create a link to a note (laurent22#5629) commit 33249ca Author: Laurent Cozic <[email protected]> Date: Mon Oct 25 19:46:45 2021 +0100 All: Improved handling of expired sessions when using Joplin Server commit ace1118 Author: Laurent Cozic <[email protected]> Date: Mon Oct 25 17:49:38 2021 +0100 All: Improved handling of expired sessions when using Joplin Server commit b497177 Author: Laurent Cozic <[email protected]> Date: Mon Oct 25 17:36:40 2021 +0100 Server: Delete all sessions when a password is changed or reset commit 82227b2 Author: Laurent Cozic <[email protected]> Date: Mon Oct 25 16:49:21 2021 +0100 Doc: Improved caching of asset URLs commit 2d85363 Author: Laurent Cozic <[email protected]> Date: Mon Oct 25 16:24:14 2021 +0100 Update website commit 6714b37 Author: Laurent Cozic <[email protected]> Date: Mon Oct 25 16:22:16 2021 +0100 Doc: Added Hosting.de sponsor commit 1a90ad3 Author: Piotr Kowalski <[email protected]> Date: Mon Oct 25 13:31:01 2021 +0200 Server: Fixed display of latest migration in startup log (laurent22#5627) commit 17cf9b3 Author: Laurent Cozic <[email protected]> Date: Sun Oct 24 19:13:56 2021 +0100 Tools: Improved desktop app runForTesting script commit 22e5c3a Author: Laurent Cozic <[email protected]> Date: Sun Oct 24 19:12:25 2021 +0100 Desktop: Fixed Goto Anything scrolling for long lists commit 72c1235 Author: Laurent Cozic <[email protected]> Date: Sun Oct 24 14:00:44 2021 +0100 All: Improved Joplin Server configuration check to better handle disabled accounts commit 2db886a Author: Roman Musin <[email protected]> Date: Sun Oct 24 13:02:37 2021 +0100 Doc: Update node version in BUILD.md (laurent22#5616) * Update node version in BUILD.md Apparently node 16.0.0 may not work, see laurent22#5608 * Update BUILD.md * Change node 16.11 -> node 14 commit 2f09f88 Author: Laurent Cozic <[email protected]> Date: Sun Oct 24 12:22:49 2021 +0100 Server: Run oversized account task more frequently commit bc5a853 Author: Laurent Cozic <[email protected]> Date: Sun Oct 24 12:17:59 2021 +0100 Server: Improved task service log entries commit 643bddf Author: Laurent Cozic <[email protected]> Date: Sun Oct 24 11:37:16 2021 +0100 Added comments commit d4a0322 Author: Laurent Cozic <[email protected]> Date: Sat Oct 23 21:58:53 2021 +0100 Server v2.5.5 commit 82f7052 Author: Laurent Cozic <[email protected]> Date: Sat Oct 23 21:57:45 2021 +0100 Tools: Fixing server Docker image build commit cbbaad9 Author: Laurent Cozic <[email protected]> Date: Sat Oct 23 20:10:04 2021 +0100 Server v2.5.4 commit 4659d57 Author: Laurent Cozic <[email protected]> Date: Sat Oct 23 20:08:56 2021 +0100 Tools: Fixing server Docker image build commit 46a4ed0 Author: Laurent Cozic <[email protected]> Date: Sat Oct 23 18:23:45 2021 +0100 Server v2.5.3 commit 56454f4 Author: Laurent Cozic <[email protected]> Date: Sat Oct 23 18:06:14 2021 +0100 Doc: Updated Font Awesome and added link to Mastodon feed commit 169b585 Author: Laurent Cozic <[email protected]> Date: Sat Oct 23 17:51:44 2021 +0100 Server: Added tool to delete old changes commit a75eba1 Author: WestfalenYeti <[email protected]> Date: Fri Oct 22 16:21:39 2021 +0200 Linux: Fixes laurent22#5568: Adds 'sort' to pre-release version list (laurent22#5588) Added a 'sort by version number' to the handling of the version list when prerelease parameter ist set. (Assumes the latest pre-release version has a highest version number) commit 375214a Author: sleepntsheep <[email protected]> Date: Fri Oct 22 00:56:40 2021 +0700 All: Translation: Update th_TH.po (laurent22#5609) Co-authored-by: sleepntsheep <[email protected]> commit 2a004dd Author: Helmut K. C. Tessarek <[email protected]> Date: Wed Oct 20 11:56:03 2021 -0400 All: Translation: Update da_DK.po (thanks ERYpTION) commit 843e407 Author: Laurent Cozic <[email protected]> Date: Tue Oct 19 17:51:15 2021 +0100 Doc: Added complete logo in different formats commit 04b8204 Author: Laurent Cozic <[email protected]> Date: Tue Oct 19 12:53:11 2021 +0100 android-2.5.1 commit 0188824 Merge: 6eced7e 72db8e4 Author: Laurent Cozic <[email protected]> Date: Tue Oct 19 11:13:38 2021 +0100 Merge branch 'dev' into release-2.5 commit 72db8e4 Author: Laurent Cozic <[email protected]> Date: Mon Oct 18 12:37:25 2021 +0100 All: Added mechanism to migrate default settings to new values commit 7d62df8 Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 19:08:34 2021 +0100 Desktop: Improved master password state handling in Encryption screen commit 6eced7e Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 18:52:35 2021 +0100 Desktop release v2.5.4 commit 8d09ed3 Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 18:50:56 2021 +0100 Fixed package path commit b891915 Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 17:45:10 2021 +0100 Desktop release v2.5.3 commit 1406d97 Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 17:20:59 2021 +0100 Mobile: Fixes laurent22#5585: Fixed logic of setting master password in Encryption screen commit a5560a6 Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 16:13:12 2021 +0100 Fixed mobile package commit 21a7149 Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 14:54:48 2021 +0100 Fixed mobile build commit 176d328 Author: Laurent Cozic <[email protected]> Date: Sun Oct 17 14:50:35 2021 +0100 ignore files commit 6bcd801 Author: Laurent Cozic <[email protected]> Date: Sat Oct 16 12:19:19 2021 +0100 Fixed interface commit 5e6e1bf Author: Kingsley Yung <[email protected]> Date: Sat Oct 16 19:19:53 2021 +0800 Cli: Fixes laurent22#5341: Ignore newline between quotes while spliting batch (laurent22#5540) commit 1d46d9f Author: Kenichi Kobayashi <[email protected]> Date: Sat Oct 16 19:44:02 2021 +0900 Desktop: Resolves laurent22#4827: Laggy scrolling in Markdown viewer (laurent22#5496) commit 6879481 Author: Roman Musin <[email protected]> Date: Sat Oct 16 10:07:41 2021 +0100 Desktop: Resolves laurent22#5168: Add support for callback URLs (laurent22#5416) commit 4322acc Author: Filip Stanis <[email protected]> Date: Sat Oct 16 10:04:53 2021 +0100 Doc: Update link to React Native docs (laurent22#5570) commit 2acd55e Author: Caleb John <[email protected]> Date: Sat Oct 16 01:59:37 2021 -0700 Desktop, Cli: Resolves laurent22#5224: Add Markdown + Front Matter exporter/importer (laurent22#5465)

CalebJohn marked this pull request as draft September 15, 2021 06:34

laurent22 marked this pull request as ready for review September 15, 2021 08:05

laurent22 reviewed Sep 15, 2021

View reviewed changes

CalebJohn marked this pull request as draft September 15, 2021 17:03

CalebJohn added the desktop All desktop platforms label Sep 19, 2021

CalebJohn added 4 commits September 27, 2021 00:40

Add Yaml exporter/importer

794bd60

This new system inherits from the MD exporter/importer but uses a yaml metadata block at the start of a note to store useful metadata

Use local time in yaml import tests

c92b804

await while adding tags Time -> time make importedNotes private again

Add tests for Yaml Export

5893bb7

Sort exported tags Adjust boolean field to comply with FAILSAFE yaml

Fix quoting issues with negative numbers

785d6d0

Fix arbitrary date format import Add extra tests

CalebJohn force-pushed the yaml-export branch from 91d14dd to 785d6d0 Compare September 27, 2021 07:41

Fix the package-lock version change

524c97c

CalebJohn marked this pull request as ready for review September 27, 2021 07:45

Merge branch 'dev' into yaml-export

c19c9d9

laurent22 reviewed Sep 28, 2021

View reviewed changes

CalebJohn added 11 commits September 28, 2021 20:35

rename Yaml export to Markdown + Metadata

0de144a

Add importing support for multimarkdown, r-markdown and pandoc

9ad9abc

Fix error with time being return in utc rather than local

Markdown + Metadata -> Markdown + Front Matter

d76b810

Add partial support for Jekyll and hugo import

893ae2f

removed category field from being used as tags (it's more akin to folder) Added lastmod for hugo handling https://jekyllrb.com/docs/front-matter/ https://gohugo.io/content-management/front-matter

Re-order ignore files

c67f759

Merge branch 'dev' into yaml-export

e0fe889

Merge branch 'dev' into yaml-export

bf39290

Merge branch 'dev' into yaml-export

74b4e44

Add test for importing notes with timezone information

96beac1

Don't export coordinates if they're all 0

7469530

This case happens when geo location is turned off. So there's no information to export

CalebJohn added 3 commits October 5, 2021 20:46

Use lowercase for field names

2967a7d

This ensures greater overlap with external tools (e.g. pandoc)

Use Iso 8601 RFC 3339 datetime for export

e7af96a

This ensures datetimes are specified in UTC

Change display of MD_FRONTMATTER to MD

f3db0bc

This is a universal change so that only the part in front of the _ will be displayed in the GUI, the assumption being that the description will differentiate the formats

laurent22 reviewed Oct 7, 2021

View reviewed changes

Add spec document

5cfd203

Correct some capitalization issues in the Exporter

Fix failing test No -> no

24daa95

laurent22 reviewed Oct 8, 2021

View reviewed changes

Add more comprehensive types

b01d43a

laurent22 merged commit 2acd55e into laurent22:dev Oct 16, 2021

CalebJohn deleted the yaml-export branch October 16, 2021 17:01

runchard pushed a commit to runchard/joplin that referenced this pull request Oct 17, 2021

Desktop, Cli: Resolves laurent22#5224: Add Markdown + Front Matter ex…

d437e77

…porter/importer (laurent22#5465)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Desktop: Resolves #5224: Add Yaml exporter/importer #5465

Desktop: Resolves #5224: Add Yaml exporter/importer #5465

CalebJohn commented Sep 15, 2021

laurent22 commented Sep 15, 2021

laurent22 left a comment

laurent22 Sep 15, 2021

laurent22 commented Sep 15, 2021

CalebJohn commented Sep 16, 2021

laurent22 commented Sep 18, 2021

CalebJohn commented Sep 27, 2021

laurent22 left a comment

laurent22 Sep 28, 2021

CalebJohn Sep 28, 2021

laurent22 Sep 29, 2021

CalebJohn Sep 29, 2021

laurent22 left a comment

laurent22 Oct 7, 2021

CalebJohn Oct 7, 2021

laurent22 Oct 8, 2021

CalebJohn commented Oct 7, 2021

laurent22 left a comment

laurent22 Oct 8, 2021

laurent22 commented Oct 16, 2021

Desktop: Resolves #5224: Add Yaml exporter/importer #5465

Desktop: Resolves #5224: Add Yaml exporter/importer #5465

Conversation

CalebJohn commented Sep 15, 2021

laurent22 commented Sep 15, 2021

laurent22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurent22 commented Sep 15, 2021

CalebJohn commented Sep 16, 2021

laurent22 commented Sep 18, 2021

CalebJohn commented Sep 27, 2021

laurent22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurent22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CalebJohn commented Oct 7, 2021

laurent22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurent22 commented Oct 16, 2021