Elasticsearch: Request Entity Too Large

[MarkusAmshove]

Description

I've tried to enable code indexing in our instance using Elasticsearch, but I get the following error for a lot of repositories:

workergroup.go:102:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 20 items, backoff for a few seconds
indexer.go:128:func2() [E] Codes indexer handler: index error for repo 713: elastic: Error 413 (Request Entity Too Large)
workergroup.go:102:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds

I've changed the setting http.max_content_length in the Elasticsearch config to the maximum possile value 2147483647b but the error still comes up.

This also comes up for a lot of repositories, not just our biggest ones.

I'm unsure how the indexer works, does it take the whole sourcecode of a branch and pumps it into elastic? Is some kind of batching per x files needed?

Gitea Version

1.21.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Running Gitea on Linux amd64 with the official binary and Elasticsearch within Docker

Database

None

[MarkusAmshove]

The repository sizes (as reported in the gitea web ui) from some repositories that I picked out of the log are:

• 1.1 MiB
• 7.8 MiB
• 2 MiB
• 598 MiB
• 154 MiB
• 2.8 MiB

That makes me wonder if the small repositories are batched together with the big ones which then exceeds the request limit.

Reducing the max file size to MAX_FILE_SIZE=10000 does not seem to resolve the issue.

[inferno-umar]

Me 2 having the same issue.
My repo size in UI is 319MiB not indexing

[wxiaoguang]

to inferno-umar : Does this answer help?

https://stackoverflow.com/questions/58490210/the-remote-server-returned-an-error-413-request-entity-too-large-elasticsear

---

update: MarkusAmshove's report said that they have tried http.max_content_length, I am wondering whether these problems are the same.

[wxiaoguang]

Unfortunately, after a quick look, I think your guess is right ... maybe Gitea does put everything into one request and send it to elasticsearch, since the first elasticsearch PR: #10273

gitea/modules/indexer/code/elasticsearch/elasticsearch.go

Lines 182 to 188 in 688d4a1

	if len(reqs) > 0 {
	_, err := b.inner.Client.Bulk().
	Index(b.inner.VersionedIndexName()).
	Add(reqs...).
	Do(ctx)
	return err
	}

[inferno-umar]

Unfortunately, after a quick look, I think your guess is right ... maybe Gitea does put everything into one request and send it to elasticsearch, since the first elasticsearch PR: #10273

gitea/modules/indexer/code/elasticsearch/elasticsearch.go

Lines 182 to 188 in 688d4a1

	if len(reqs) > 0 {
	_, err := b.inner.Client.Bulk().
	Index(b.inner.VersionedIndexName()).
	Add(reqs...).
	Do(ctx)
	return err
	}

Yeah! you're right Gitea is putting everything in 1 request before sending it to elastic search, not batching it, as shown in my error logs below:

....
...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 28: elastic: Error 413 (Request Entity Too Large)
...queue/workergroup.go:102:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
.....

[inferno-umar]

After finding out I pushed my elasticsearch maximum limit to 2147483647b then I'm getting the following error 429 (too many requests):

 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 28: elastic: Error 429 (Too Many Requests): [in_flight_requests] Data too large, data for [<http_request>] would be  [1272753908/1.1gb], which is larger than the limit of [1073741824/1gb] [type=circuit_breaking_exception]

[inferno-umar]

I'm trying to fix this issue in the code by batching the requests

Unfortunately, after a quick look, I think your guess is right ... maybe Gitea does put everything into one request and send it to elasticsearch, since the first elasticsearch PR: #10273

gitea/modules/indexer/code/elasticsearch/elasticsearch.go

Lines 182 to 188 in 688d4a1

	if len(reqs) > 0 {
	_, err := b.inner.Client.Bulk().
	Index(b.inner.VersionedIndexName()).
	Add(reqs...).
	Do(ctx)
	return err
	}

[lunny]

Fixed by #29075

[github-actions]

Automatically locked because of our CONTRIBUTING guidelines