docs: sync Haystack API with Readme

.github/utils/pydoc-markdown.sh

@@ -1,7 +1,10 @@
 #!/bin/bash
 
 set -e # Fails on any error in the following loop
+export PYTHONPATH=$PWD/docs/pydoc # Make the renderers available to pydoc
 cd docs/_src/api/api/
-for file in ../pydoc/* ; do
+mkdir temp
+cd temp
+for file in ../../pydoc/* ; do
  pydoc-markdown "$file"
 done

.github/workflows/readme_integration.yml

@@ -0,0 +1,34 @@
+name: Testing Haystack-Readme integration
+
+on:
+ push:
+ branches:
+ - readme_integration_api
+
+jobs:
+ sync:
+ runs-on: ubuntu-latest
+ steps:
+
+ - name: Checkout this repo
+ uses: actions/checkout@v3
+
+ - name: Set up Python 3.8.10
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.8.10
+
+ - name: Install dependencies
+ run: |
+ python -m pip install --upgrade pip
+ pip install pydoc-markdown==4.5.1
+
+ - name: Generate API docs
+ run: ./.github/utils/pydoc-markdown.sh
+
+ - name: Github Action (API sync)
+ uses: readmeio/[email protected]
+ env:
+ README_API_KEY: ${{ secrets.README_API_KEY }}
+ with:
+ rdme: docs ./docs/_src/api/api/temp --key="$README_API_KEY" --version=1.8-latest

docs/_src/api/api/file_converter.md

@@ -288,7 +288,7 @@ def convert(file_path: Path,
  id_hash_keys: Optional[List[str]] = None) -> List[Document]
 ```
 
-Reads text from a txt file and executes optional preprocessing steps.
+Reads text from a markdown file and executes optional preprocessing steps.
 
 **Arguments**:
 

docs/_src/api/api/primitives.md

@@ -144,9 +144,9 @@ class Span()
 
 #### end
 
-Defining a sequence of characters (Text span) or cells (Table span) via start and end index. 
+Defining a sequence of characters (Text span) or cells (Table span) via start and end index.
 
-For extractive QA: Character where answer starts/ends 
+For extractive QA: Character where answer starts/ends
 For TableQA: Cell where the answer starts/ends (counted from top left to bottom right of table)
 
 **Arguments**:
@@ -174,24 +174,24 @@ For example, it's used within some Nodes like the Reader, but also in the REST A
 **Arguments**:
 
 - `answer`: The answer string. If there's no possible answer (aka "no_answer" or "is_impossible) this will be an empty string.
-- `type`: One of ("generative", "extractive", "other"): Whether this answer comes from an extractive model 
-(i.e. we can locate an exact answer string in one of the documents) or from a generative model 
+- `type`: One of ("generative", "extractive", "other"): Whether this answer comes from an extractive model
+(i.e. we can locate an exact answer string in one of the documents) or from a generative model
 (i.e. no pointer to a specific document, no offsets ...).
 - `score`: The relevance score of the Answer determined by a model (e.g. Reader or Generator).
 In the range of [0,1], where 1 means extremely relevant.
 - `context`: The related content that was used to create the answer (i.e. a text passage, part of a table, image ...)
 - `offsets_in_document`: List of `Span` objects with start and end positions of the answer **in the
 document** (as stored in the document store).
-For extractive QA: Character where answer starts => `Answer.offsets_in_document[0].start 
+For extractive QA: Character where answer starts => `Answer.offsets_in_document[0].start
 For TableQA: Cell where the answer starts (counted from top left to bottom right of table) => `Answer.offsets_in_document[0].start
 (Note that in TableQA there can be multiple cell ranges that are relevant for the answer, thus there can be multiple `Spans` here)
 - `offsets_in_context`: List of `Span` objects with start and end positions of the answer **in the
 context** (i.e. the surrounding text/table of a certain window size).
-For extractive QA: Character where answer starts => `Answer.offsets_in_document[0].start 
+For extractive QA: Character where answer starts => `Answer.offsets_in_document[0].start
 For TableQA: Cell where the answer starts (counted from top left to bottom right of table) => `Answer.offsets_in_document[0].start
 (Note that in TableQA there can be multiple cell ranges that are relevant for the answer, thus there can be multiple `Spans` here)
 - `document_id`: ID of the document that the answer was located it (if any)
-- `meta`: Dict that can be used to associate any kind of custom meta data with the answer. 
+- `meta`: Dict that can be used to associate any kind of custom meta data with the answer.
 In extractive QA, this will carry the meta data of the document where the answer was found.
 
 <a id="schema.Answer.__lt__"></a>

docs/_src/api/pydoc/answer-generator.yml

@@ -12,9 +12,17 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: generator.md
+ type: renderers.ReadmeRenderer
+ excerpt: Reads a set of documents and generates an answer to a question, word by word
+ category: 6310ca73c622850ddd3875a2
+ title: Answer Generator API
+ slug: answer-generator-api
+ order: 0
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: answer_generator_api.md
+
+

docs/_src/api/pydoc/crawler.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: crawler.md
+ type: renderers.ReadmeRenderer
+ excerpt: The Crawler scrapes the text from a website, creates a Haystack Document object out of it, and saves it to a JSON file.
+ category: 6310ca73c622850ddd3875a2
+ title: Crawler API
+ slug: crawler-api
+ order: 10
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: crawler_api.md

docs/_src/api/pydoc/document-classifier.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: document_classifier.md
+ type: renderers.ReadmeRenderer
+ excerpt: Used to create predictions that are attached to documents as metadata.
+ category: 6310ca73c622850ddd3875a2
+ title: Document Classifier API
+ slug: document-classifier-api
+ order: 20
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: document_classifier_api.md

docs/_src/api/pydoc/document-store.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: document_store.md
+ type: renderers.ReadmeRenderer
+ excerpt: Stores your texts and meta data and provides them to the Retriever at query time.
+ category: 6310ca73c622850ddd3875a2
+ title: Document Store API
+ slug: document-store-api
+ order: 30
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: document_store_api.md

docs/_src/api/pydoc/evaluation.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: evaluation.md
+ type: renderers.ReadmeRenderer
+ excerpt: The tools needed to evaluate whole pipelines or individual Nodes.
+ category: 6310ca73c622850ddd3875a2
+ title: Evaluation API
+ slug: evaluation-api
+ order: 40
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: evaluation_api.md

docs/_src/api/pydoc/extractor.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: extractor.md
+ type: renderers.ReadmeRenderer
+ excerpt: Extracts predefined entities out of a piece of text.
+ category: 6310ca73c622850ddd3875a2
+ title: Entity Extractor API
+ slug: entity-extractor-api
+ order: 50
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: entity_extractor_api.md

docs/_src/api/pydoc/file-classifier.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: file_classifier.md
+ type: renderers.ReadmeRenderer
+ excerpt: Distinguishes between text, PDF, Markdown, Docx and HTML files and routes them to the appropriate File Converter in an indexing pipeline.
+ category: 6310ca73c622850ddd3875a2
+ title: File Classifier API
+ slug: file-classifier-api
+ order: 60
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: file_classifier_api.md

docs/_src/api/pydoc/file-converters.yml

@@ -5,16 +5,22 @@ loaders:
  ignore_when_discovered: ['__init__']
 processors:
  - type: filter
- expression:
+ expression: 
  documented_only: true
  do_not_filter_modules: false
  skip_empty_modules: true
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: file_converter.md
+ type: renderers.ReadmeRenderer
+ excerpt: Extracts text from files in different formats and cast it into the unified Document format.
+ category: 6310ca73c622850ddd3875a2
+ title: File Converters API
+ slug: file-converters-api
+ order: 70
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: file_converters_api.md

docs/_src/api/pydoc/other-nodes.yml → docs/_src/api/pydoc/other.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: other_nodes.md
+ type: renderers.ReadmeRenderer
+ excerpt: The utility classes of Haystack.
+ category: 6310ca73c622850ddd3875a2
+ title: Other API
+ slug: other-api
+ order: 80
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: other_api.md

docs/_src/api/pydoc/pipelines.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: pipelines.md
+ type: renderers.ReadmeRenderer
+ excerpt: Arranges nodes in a predefined flow.
+ category: 6310ca73c622850ddd3875a2
+ title: Pipelines API
+ slug: pipelines-api
+ order: 90
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: pipelines_api.md

docs/_src/api/pydoc/preprocessor.yml

@@ -12,9 +12,15 @@ processors:
  - type: smart
  - type: crossref
 renderer:
- type: markdown
- descriptive_class_title: false
- descriptive_module_title: true
- add_method_class_prefix: true
- add_member_class_prefix: false
- filename: preprocessor.md
+ type: renderers.ReadmeRenderer
+ excerpt: Normalize white spaces, gets rid of headers and footers, cleans empty lines in your Documents, or splits them into smaller pieces.
+ category: 6310ca73c622850ddd3875a2
+ title: PreProcessor API
+ slug: preprocessor-api
+ order: 100
+ markdown:
+ descriptive_class_title: false
+ descriptive_module_title: true
+ add_method_class_prefix: true
+ add_member_class_prefix: false
+ filename: preprocessor_api.md