core[patch], langchain[patch]: ByteStore (#14312)

docs/.local_build.sh

@@ -9,7 +9,7 @@ SCRIPT_DIR="$(cd "$(dirname "$0")"; pwd)"
 cd "${SCRIPT_DIR}"
 
 mkdir -p ../_dist
-rsync -ruv --exclude node_modules . ../_dist
+rsync -ruv --exclude node_modules --exclude api_reference --exclude .venv --exclude .docusaurus . ../_dist
 cd ../_dist
 poetry run python scripts/model_feat_table.py
 cp ../cookbook/README.md src/pages/cookbook.mdx

docs/docs/integrations/providers/redis.mdx

@@ -17,6 +17,24 @@ Install the Python SDK:
 pip install redis
 ```
 
+To run Redis locally, you can use Docker:
+
+```bash
+docker run --name langchain-redis -d -p 6379:6379 redis redis-server --save 60 1 --loglevel warning
+```
+
+To stop the container:
+
+```bash
+docker stop langchain-redis
+```
+
+And to start it again:
+
+```bash
+docker start langchain-redis
+```
+
 ## Wrappers
 
 All wrappers need a redis url connection string to connect to the database support either a stand alone Redis server

docs/docs/integrations/stores/file_system.ipynb

@@ -0,0 +1,100 @@
+{
+ "cells": [
+ {
+ "cell_type": "raw",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "sidebar_label: Local Filesystem\n",
+ "sidebar_position: 3\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# LocalFileStore\n",
+ "\n",
+ "The `LocalFileStore` is a persistent implementation of `ByteStore` that stores everything in a folder of your choosing."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[b'v1', b'v2']\n"
+ ]
+ }
+ ],
+ "source": [
+ "from pathlib import Path\n",
+ "\n",
+ "from langchain.storage import LocalFileStore\n",
+ "\n",
+ "root_path = Path.cwd() / \"data\" # can also be a path set by a string\n",
+ "store = LocalFileStore(root_path)\n",
+ "\n",
+ "store.mset([(\"k1\", b\"v1\"), (\"k2\", b\"v2\")])\n",
+ "print(store.mget([\"k1\", \"k2\"]))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now let's see which files exist in our `data` folder:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "k1 k2\n"
+ ]
+ }
+ ],
+ "source": [
+ "!ls {root_path}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": ".venv",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/docs/integrations/stores/in_memory.ipynb

@@ -0,0 +1,73 @@
+{
+ "cells": [
+ {
+ "cell_type": "raw",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "sidebar_label: In Memory\n",
+ "sidebar_position: 2\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# InMemoryByteStore\n",
+ "\n",
+ "The `InMemoryByteStore` is a non-persistent implementation of `ByteStore` that stores everything in a Python dictionary."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[b'v1', b'v2']\n"
+ ]
+ }
+ ],
+ "source": [
+ "from langchain.storage import InMemoryByteStore\n",
+ "\n",
+ "store = InMemoryByteStore()\n",
+ "\n",
+ "store.mset([(\"k1\", b\"v1\"), (\"k2\", b\"v2\")])\n",
+ "print(store.mget([\"k1\", \"k2\"]))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": ".venv",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/docs/integrations/stores/index.mdx

@@ -0,0 +1,29 @@
+---
+sidebar_position: 1
+sidebar_class_name: hidden
+---
+
+# Stores
+
+In many different applications, having some sort of key-value storage is helpful. 
+In this section, we will look at a few different ways to store key-value pairs
+using implementations of the `ByteStore` interface.
+
+## Features (natively supported)
+
+All `ByteStore`s support the following functions, which are used for modifying
+**m**ultiple key-value pairs at once:
+
+- `mget(key: Sequence[str]) -> List[Optional[bytes]]`: get the contents of multiple keys, returning `None` if the key does not exist
+- `mset(key_value_pairs: Sequence[Tuple[str, bytes]]) -> None`: set the contents of multiple keys
+- `mdelete(key: Sequence[str]) -> None`: delete multiple keys
+- `yield_keys(prefix: Optional[str] = None) -> Iterator[str]`: yield all keys in the store, optionally filtering by a prefix
+
+## How to pick one
+
+`ByteStore`s are designed to be interchangeable. By default, most dependent integrations
+use the `InMemoryByteStore`, which is a simple in-memory key-value store.
+
+However, if you start having other requirements, like massive scalability or persistence,
+you can swap out the `ByteStore` implementation with one of the other ones documented
+in this section.

docs/docs/integrations/stores/redis.ipynb

@@ -0,0 +1,83 @@
+{
+ "cells": [
+ {
+ "cell_type": "raw",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "sidebar_label: Redis\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# RedisStore\n",
+ "\n",
+ "The `RedisStore` is an implementation of `ByteStore` that stores everything in your Redis instance.\n",
+ "\n",
+ "To configure Redis, follow our [Redis guide](/docs/integrations/providers/redis)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!pip install redis"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[b'v1', b'v2']\n"
+ ]
+ }
+ ],
+ "source": [
+ "from langchain.storage import RedisStore\n",
+ "\n",
+ "store = RedisStore(redis_url=\"redis:https://localhost:6379\")\n",
+ "\n",
+ "store.mset([(\"k1\", b\"v1\"), (\"k2\", b\"v2\")])\n",
+ "print(store.mget([\"k1\", \"k2\"]))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": ".venv",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}