refactor: WordPress data import #506

guqing · 2020-01-18T14:03:44Z

使用xml转json的方式替代原来的xml递归解析为map
新增转换器用于将wordpress的博客数据对象转为PostVo方便保存
该方式用于是将先将博客导出的xml数据转为json在映射到对象属性封装,缺点是转两次,但使用xml解析比如Jaxb的方式没有json灵活甚至由于名称空间问题有些属性无法解析,且二者效率上差别不大,且都逃避不了需要自定义多个entity对象来接受xml的属性数据

JohnNiang · 2020-01-18T14:10:55Z

build.gradle

@@ -104,6 +107,10 @@ dependencies {
 implementation "net.sf.image4j:image4j:$image4jVersion"
 implementation "org.flywaydb:flyway-core:$flywayVersion"

+ implementation "org.json:json:$jsonVersion"
+ implementation "commons-io:commons-io:$commonsIOVersion"
+ implementation "com.alibaba:fastjson:$fastJsonVersion"


你可以考虑使用 run.halo.app.utils.JsonUtils，而不必要引入 fastjson。

我试试你说的@JsonProperty如果可以实现我就把fastjson删了😂

不建议再引入 fastjson。另外，这个 commons-io 似乎也有。

commonsio已经排除但是fastjson不行，使用halo的JsonUtils配合@JsonProperty无法将json属性映射到对象

JohnNiang · 2020-01-18T14:12:54Z

src/main/java/run/halo/app/handler/migrate/WordPressMigrateHandler.java

@@ -90,136 +84,18 @@ public WordPressMigrateHandler(AttachmentService attachmentService,
 public void migrate(MultipartFile file) {


这里的参数以后考虑更改了，不能使用 MultipartFile file。或者创建一个适配器来兼容它。

这里老汪写的我沿用了

这里老汪写的我沿用了

嗯嗯，这不是你的锅，是我们当时设计的时候没有考虑好。

JohnNiang · 2020-01-18T14:34:02Z

src/main/java/run/halo/app/handler/migrate/converter/Converter.java

+ * @author guqing
+ * @date 2020-01-18 16:45
+ */
+public class Converter<T, U> {


建议此 Convert 支持双向转换：P

之前支持但是我想了一下并不需要将数据库的文章映射到wordpress的实体如果添加双向需要在构造里在写一个Function并不好实现很累赘所以就删除了

之前支持但是我想了一下并不需要将数据库的文章映射到wordpress的实体如果添加双向需要在构造里在写一个Function并不好实现很累赘所以就删除了

只是一个临时的想法，哈哈哈，不过建议预留这个接口。

public class Converter<T, U> { // 从 T 转换为 U private Function<T, U> fromDto; // 从 U 转换为 T private Function<U, T> fromEntity; public Converter(final Function<T, U> fromDto, final Function<U, T> fromEntity) { this.fromDto = fromDto; this.fromEntity = fromEntity; } public final U converterFromDto(final T dto){ return fromDto.apply(dto); } public final T converterFromEntity(final U entity){ return fromEntity.apply(entity); } public final List<U> batchConverterFromDto(final List<T> dtos){ return dtos.stream().map(this::converterFromDto).collect(Collectors.toList()); } public final List<T> batchConverterFromEntity(final List<U> entities){ return entities.stream().map(this::converterFromEntity).collect(Collectors.toList()); } }

之前是这么写的, 如果用得上后面也可以,现在添加也行, 除非有人想把halo的数据迁移到wordpress,哈哈哈

建议将 Converter 声明为接口，可以提供一些默认的实现。

好，我改一下

ruibaby · 2020-01-18T15:41:51Z

build.gradle

@@ -104,6 +107,10 @@ dependencies {
 implementation "net.sf.image4j:image4j:$image4jVersion"
 implementation "org.flywaydb:flyway-core:$flywayVersion"

+ implementation "org.json:json:$jsonVersion"
+ implementation "commons-io:commons-io:$commonsIOVersion"
+ implementation "com.alibaba:fastjson:$fastJsonVersion"


不建议再引入 fastjson。另外，这个 commons-io 似乎也有。

ruibaby · 2020-01-18T15:43:30Z

src/main/java/run/halo/app/handler/migrate/support/wordpress/Channel.java

+ * @date 2019-11-17 13:57
+ */
+@Data
+public class Channel {


这里最好给每个属性加上 java doc 注释，后面 Wordpress 可能会改一些东西，免得后面不清楚。

ruibaby · 2020-01-18T15:43:40Z

src/main/java/run/halo/app/handler/migrate/support/wordpress/Comment.java

+ * @date 2019-11-17 14:01
+ */
+@Data
+public class Comment {


甚至可以链接我们对应的类和属性。

ruibaby · 2020-01-18T15:44:22Z

src/main/java/run/halo/app/handler/migrate/support/wordpress/Item.java

+ * @date 2019-11-17 13:59
+ */
+@Data
+public class Item {


同上，加上 java doc。

ruibaby · 2020-01-18T15:45:23Z

src/test/resources/wordpressdemo.xml

@@ -0,0 +1,144 @@
+<?xml version="1.0" encoding="UTF-8" ?>


这个可能是不完整的，后面我会再导出一个，用来测试，或者叫群友提供。

好现在只是转了然后覆盖了你migrateHandle里的写法还没有写存储逻辑但是先将xml转json一般不会有什么问题可能出现的问题只是我自定义的映射entity字段没写全不过我今晚转换数据库实体的时候已经没有缺少的字段了都能补全后面在导几个来测试一下没问题在写入库

…ss convert null judgment

… It can be used to mapping the relationship between source model and target model

…tom annotation

src/main/java/run/halo/app/handler/migrate/converter/Converter.java

JohnNiang · 2020-01-19T09:27:47Z

src/main/java/run/halo/app/handler/migrate/converter/WordPressConverter.java

- BasePost post = getBasePostFromItem(item);
- postVo.setBasePost(post);
+ List<Item> items = channel.getItems();
+ Assert.notNull(items, "解析出错,文章item项不能为null");


We do not recommend the use of Assert#notNull from here. Because the items is not a parameter, and this check may throw java.lang.IllegalArgumentException.

挨不对啊这里我已经改过了难道没同步过去
WordPressConvert

* feat: support generate static pages. * feat: support copy static folder when generate static pages. * feat: clean static pages folder before generate. * feat: add static deploy handlers. * feat: support deploy static page in netlify. * feat: support replace url method. * feat: add data process apis. * feat: custom permalink. * feat: support set path suffix. * feat: create get by date api for post. * feat: support /year/month/day/url and /year/month/url mapping. * feat: support set categories and tags mapping prefix. * feat: add leveldb cache store impl (#494) add leveldb config by yaml dev add cache impl test * feat:(add comment blacklist) (#465) * feat:(add comment blacklist) * update enum name * update enum * fix:(update pom) * 优化相关的细节 * none: add some test case, but ignored * Disable url regex match temporarily * Fix vulnerability due to uncheck theme id (#499) * fix: #489 * Update FUNDING.yml * merge from master * feat: update page default sort && update the pre and next of posts * feat: adapter dev branch * Feat: update getAdjacentPosts return && update post ftl * fix: revert gradle-wrapper.jar Co-authored-by: Ryan Wang <[email protected]> Co-authored-by: Jiahuan Shen <[email protected]> Co-authored-by: John Niang <[email protected]> Co-authored-by: weiwensangsang <[email protected]> Co-authored-by: Lei XinXin <[email protected]> Co-authored-by: 寒山 <[email protected]> Co-authored-by: IJKZEN <[email protected]> * feat: create theme content api. (#503) * feat: create freemarker custom tag of tools. (#505) * feat: create freemarker custom tag of tools. * chore: update dependencies. (#510) * chore: update dependencies. * refactor: WordPress data import (#506) * refactor: WordPress data import * refactor: change wordpress convert class to interface and add wordpress convert null judgment * fix: add some class doc * fix: exclude commons io * feat: add custom annotation: @PropertyMappingTo and reflection utils, It can be used to mapping the relationship between source model and target model * feat: replace the original wordpressconvert implementation with a custom annotation * fix: change wordpress convert method name * fix: change wordpress convert method name * fix: merger conflict * Enable github ci on pull request * Enable github ci on pull request (#525) * Add haloCodeStyle.xml (#524) * Uncomment content url mappings (#523) * Close file input stream after reading image file (#527) * Fix test error (#528) * Enable github action cache for CI * feat: support set theme static source path type. (#535) * feat: support set theme static source path type. * refactor: global_absolute_path_enabled property. * Fix wordpress importor. (#536) * feat: support freemarker template Inheritance #534 (#537) * feat: #534 * fix: HashMap initialCapacity * feat: change themes assets base path (#538) * feat: change theme resources base path. * feat: add new variable about theme base path. * style: collate configuration files. (#540) * feat: support set attachment path type. (#539) * feat: support theme require halo version. (#544) * feat: support theme require halo version. * fix: com.sun.xml.internal.ws.util does not exist error. * fix: com.sun.xml.internal.ws.util does not exist error again. * Update ThemeServiceImpl.java * feat: enable default theme submodule support. (#548) * Remove anatole theme for submodule * Enable submodule support * Enable submodule support for checkouting by github action * Configure halo.workDir into application-test.yaml (#550) * fix: Theme refresh issue #553 (#554) * fix: Incorrect sequence of tailing log file#517 (#556) * Add custom RequestMappingHandlerMapping for enable static resources access (#558) Co-authored-by: pencilso <[email protected]> Co-authored-by: Lei XinXin <[email protected]> Co-authored-by: John Niang <[email protected]> Co-authored-by: leozhou <[email protected]> Co-authored-by: Jiahuan Shen <[email protected]> Co-authored-by: weiwensangsang <[email protected]> Co-authored-by: 寒山 <[email protected]> Co-authored-by: IJKZEN <[email protected]> Co-authored-by: guqing <[email protected]>

* refactor: WordPress data import * refactor: change wordpress convert class to interface and add wordpress convert null judgment * fix: add some class doc * fix: exclude commons io * feat: add custom annotation: @PropertyMappingTo and reflection utils, It can be used to mapping the relationship between source model and target model * feat: replace the original wordpressconvert implementation with a custom annotation * fix: change wordpress convert method name * fix: change wordpress convert method name * fix: merger conflict

refactor: WordPress data import

17f68cd

guqing requested review from ruibaby and JohnNiang and removed request for ruibaby January 18, 2020 14:04

JohnNiang requested changes Jan 18, 2020

View reviewed changes

ruibaby requested changes Jan 18, 2020

View reviewed changes

guqing added 5 commits January 19, 2020 00:36

refactor: change wordpress convert class to interface and add wordpre…

541d79d

…ss convert null judgment

fix: add some class doc

80d16bc

fix: exclude commons io

19becb3

feat: add custom annotation: @PropertyMappingTo and reflection utils,…

2988fbb

… It can be used to mapping the relationship between source model and target model

feat: replace the original wordpressconvert implementation with a cus…

517ef9f

…tom annotation

JohnNiang reviewed Jan 19, 2020

View reviewed changes

ruibaby added the kind/feature Categorizes issue or PR as related to a new feature. label Jan 20, 2020

ruibaby added this to the 1.3.0 milestone Jan 20, 2020

guqing added 2 commits January 20, 2020 14:37

fix: change wordpress convert method name

19044b6

fix: change wordpress convert method name

d14f6ea

ruibaby approved these changes Jan 20, 2020

View reviewed changes

ruibaby requested a review from JohnNiang January 20, 2020 06:45

guqing added 2 commits January 20, 2020 14:55

fix: merger conflict

37f0d30

fix: merger conflict

c5dac64

JohnNiang approved these changes Jan 20, 2020

View reviewed changes

JohnNiang merged commit 61f0748 into halo-dev:dev Jan 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: WordPress data import #506

refactor: WordPress data import #506

guqing commented Jan 18, 2020

JohnNiang Jan 18, 2020

guqing Jan 18, 2020

ruibaby Jan 18, 2020

guqing Jan 18, 2020

JohnNiang Jan 18, 2020

guqing Jan 18, 2020

JohnNiang Jan 18, 2020

JohnNiang Jan 18, 2020

guqing Jan 18, 2020

JohnNiang Jan 18, 2020

guqing Jan 18, 2020

JohnNiang Jan 18, 2020

guqing Jan 18, 2020

ruibaby Jan 18, 2020

ruibaby Jan 18, 2020

guqing Jan 18, 2020

ruibaby Jan 18, 2020

ruibaby Jan 18, 2020

guqing Jan 18, 2020

ruibaby Jan 18, 2020

ruibaby Jan 18, 2020

guqing Jan 18, 2020

JohnNiang Jan 19, 2020

guqing Jan 19, 2020

		@@ -90,136 +84,18 @@ public WordPressMigrateHandler(AttachmentService attachmentService,
		public void migrate(MultipartFile file) {

refactor: WordPress data import #506

refactor: WordPress data import #506

Conversation

guqing commented Jan 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment