__init__() got an unexpected keyword argument 'chunker' while using customized chunker

[YunchengLiang]

I also tried the demo in readme, same err appears. Could you please help me?

[cachho]

Can you try with the first level of the dict removed, so just

pdf_add_config = {
    "chunk_size": 200,
    ...

Linking #251

[YunchengLiang]

Can you try with the first level of the dict removed, so just

pdf_add_config = {
    "chunk_size": 200,
    ...

Linking #251

I tried but it has the following error ...

[cachho]

I think you need to say AddConfig(chunker=**pdf_add_config). I made a PR #270 that should simplify this issue.

[YunchengLiang]

chunker=**pdf_add_config

it says it is not the correct syntax, and if i remove the **, then it would say unexpected key argument "chunker" again...

[YunchengLiang]

I think you need to say AddConfig(chunker=**pdf_add_config). I made a PR #270 that should simplify this issue.

me and my colleague found a temporary solution by having code as below,

from typing import Callable, Optional
from embedchain.config.BaseConfig import BaseConfig

class ChunkerConfig(BaseConfig):
    def __init__(
        self,
        chunk_size: Optional[int] = 4000,
        chunk_overlap: Optional[int] = 200,
        length_function: Optional[Callable[[str], int]] = len,
    ):
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap
        self.length_function = length_function


from embedchain.config.AddConfig import *

chunker = {
                "chunk_size": 200,
                "chunk_overlap": 20,
                "length_function": len
        }

cc = ChunkerConfig(**chunker)
ac = AddConfig()
ac.chunker = cc
print(ac.chunker.chunk_size)

pdf_url = 'https://www.rogers.com/cms/pdf/en/Consumer_SUG_V20.pdf' #online resources
chat_bot.add('pdf_file', pdf_url, config=ac) 

This seems to work but it generate the same number of chunks (total 55 for this file) no matter which chunk_size we config.
We indeed remember to clear the db by delete everything under it every time before we change the chunk_size paramater. But it still output the same number of chunks....