is chinese id supported ?

[shatealaboxiaowang]

Hi, i want to extract the key field information of Chinese text, so does kor.Text id supports chinese ?

[eyurtsev]

Not at the moment, it's likely not that difficult to add, likely requires creating another identifier that would be used instead of the ID when writing the prompt to the LLM

[shatealaboxiaowang]

Thank you very much for your reply, but it seems that I did not understand what you wrote. The language of ‘another identifier’ should still be English, right? Do I need to do a Chinese to English mapping?

[shatealaboxiaowang]

Like the code below? Just give each 'id' a name, and in the 'description' clearly describe the field to be extracted。

"messages": {
"id": "test",
"description": "test",
"field_01": {
"id": "id_01",
"description": "歌手名字？",
"example": []
},
"field_02": {
"id": "id_02",
"description": "专辑有哪些？",
"example": []
},
"field_03": {
"id": "id_03",
"description": "蔡依林的歌曲叫什么？",
"example": []
}
},

[eyurtsev]

Kor cannot support an ID field in Chinese right now. This could be a feature that will be added at some point.

In the meantime, you could rely on examples to improve the quality of extraction. It's unclear to what extent having an ID provided in chinese will affect the quality of the result since the language models already understand multiple languages.

[shatealaboxiaowang]

Kor cannot support an ID field in Chinese right now. This could be a feature that will be added at some point.

In the meantime, you could rely on examples to improve the quality of extraction. It's unclear to what extent having an ID provided in chinese will affect the quality of the result since the language models already understand multiple languages.

Thank you very much for your reply. I have modified it on the basis of your source code, and now it supports Chinese. The modified code is as follows:

ADD：
VALID_IDENTIFIER_PATTERN_CH = re.compile(r"[\u4e00-\u9fff]+")
Modify:
if not (VALID_IDENTIFIER_PATTERN.match(uid) or VALID_IDENTIFIER_PATTERN_CH.match(uid)):

in kor.modes
could you please help check whether it is accurate?

[eyurtsev]

If you're working with your own clone of the library and you could probably remove the VALID_IDENTIFIER check completely -- as long as the code runs without the identifier and generates the correct prompt you should be OK.

[aixiamomo]

Kor cannot support an ID field in Chinese right now. This could be a feature that will be added at some point.
In the meantime, you could rely on examples to improve the quality of extraction. It's unclear to what extent having an ID provided in chinese will affect the quality of the result since the language models already understand multiple languages.

Thank you very much for your reply. I have modified it on the basis of your source code, and now it supports Chinese. The modified code is as follows:

ADD： VALID_IDENTIFIER_PATTERN_CH = re.compile(r"[\u4e00-\u9fff]+") Modify: if not (VALID_IDENTIFIER_PATTERN.match(uid) or VALID_IDENTIFIER_PATTERN_CH.match(uid)):

in kor.modes could you please help check whether it is accurate?

from kor import nodes
nodes.VALID_IDENTIFIER_PATTERN = re.compile(r".")  # monkey patch 使支持中文identifier