llm embed-multi gives a stacktrace if the pattern matches a directory

[stoerr]

If you run this, you'll get a nasty error message:

mkdir bla.js
llm embed-multi aem -d aem.db -m ada --files . '**/*.js'

It's probably better to check whether the matched things are actually readable files, and print a warning if not. Right now you get this. And that aborts the indexing, too, right? Not quite sure whether it's better to continue. Anyway: HUGE thanks for this marvelous stuff!

  File "/opt/homebrew/bin/llm", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/llm/cli.py", line 1328, in embed_multi
    collection_obj.embed_multi(tuples(), store=store)
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/llm/embeddings.py", line 165, in embed_multi
    self.embed_multi_with_metadata(
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/llm/embeddings.py", line 189, in embed_multi_with_metadata
    batch = list(islice(iterator, batch_size))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/llm/embeddings.py", line 166, in <genexpr>
    ((id, value, None) for id, value in entries), store=store
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/llm/cli.py", line 1319, in tuples
    for row in rows:
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/click/_termui_impl.py", line 344, in generator
    for rv in self.iter:
  File "/opt/homebrew/Cellar/llm/0.10/libexec/lib/python3.11/site-packages/llm/cli.py", line 1275, in iterate_files
    content = path.read_text(encoding=encoding)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/pathlib.py", line 1058, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/pathlib.py", line 1044, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IsADirectoryError: [Errno 21] Is a directory: 'bla.js'

[simonw]

The right thing to do here might to silently skip directories, since the --files argument is only meant to work with files in the first place I don't think that would surprise anyone.

[stoerr]

Right, I agree. There is not much point in logging a warning, also in the cases I had.