Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exp/lighthorizon/build/index-batch: merge map/reduce updates to latest on feature branch #4543

Merged
merged 11 commits into from
Aug 17, 2022
Merged

exp/lighthorizon/build/index-batch: merge map/reduce updates to latest on feature branch #4543

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d5c7292
Dockerize indexer to run in AWS Batch
2opremio Aug 1, 2022
e12cc83
Parameterize job index env variable (it can't be overriden in Batch)
2opremio Aug 4, 2022
2b9009a
Merge branch 'lighthorizon' into 4475_dockerize_batch_indexer
sreuland Aug 13, 2022
70aebcc
#4475: relocated indexer image docs to exp/lighthorizon/build/index-b…
sreuland Aug 13, 2022
8d148a2
#4475: added alternative k8s batch job definitions for map/reduce
sreuland Aug 15, 2022
0ba708b
Merge remote-tracking branch 'upstream/lighthorizon' into 4475_docker…
sreuland Aug 15, 2022
b706613
#4475: fix CI build/test errors
sreuland Aug 16, 2022
65db297
#4475: fixed CI workflow index build test failure
sreuland Aug 16, 2022
78e63a7
Merge remote-tracking branch 'upstream/lighthorizon' into 4475_docker…
sreuland Aug 17, 2022
2329f3c
#4475: added docs on file backend method, pr feedback
sreuland Aug 17, 2022
806b74c
#4475: fixed go fmt warning
sreuland Aug 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
#4475: fix CI build/test errors
  • Loading branch information
@sreuland
sreuland committed Aug 16, 2022
commit b7066131e9546ab60d1e3f97283e59ae0b4f7793
4 changes: 2 additions & 2 deletions .github/workflows/horizon.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,11 +145,11 @@ jobs:

# run map job
docker run -e NETWORK_PASSPHRASE='testnet' -e JOB_INDEX_ENV=AWS_BATCH_JOB_ARRAY_INDEX -e AWS_BATCH_JOB_ARRAY_INDEX=0 -e BATCH_SIZE=64 -e FIRST_CHECKPOINT=64 \
-e WORKER_COUNT=1 -v /ledgerexport-test/:/ledgermeta -e TXMETA_SOURCE=file:https:///ledgermeta -v $PWD/index:/index -e INDEX_TARGET=file:https:///index stellar/lighthorizon-index-batch map
-e WORKER_COUNT=1 -e RUN_MODE=map -v /ledgerexport-test/:/ledgermeta -e TXMETA_SOURCE=file:https:///ledgermeta -v $PWD/index:/index -e INDEX_TARGET=file:https:///index stellar/lighthorizon-index-batch

# run reduce job
docker run -e NETWORK_PASSPHRASE='testnet' -e JOB_INDEX_ENV=AWS_BATCH_JOB_ARRAY_INDEX -e AWS_BATCH_JOB_ARRAY_INDEX=0 -e MAP_JOB_COUNT=1 -e REDUCE_JOB_COUNT=1 \
-e WORKER_COUNT=1 -v $PWD/index:/index -e INDEX_SOURCE_ROOT=file:https:///index -e INDEX_TARGET=file:https:///index stellar/lighthorizon-index-batch reduce
-e WORKER_COUNT=1 -e RUN_MODE=reduce -v $PWD/index:/index -e INDEX_SOURCE_ROOT=file:https:///index -e INDEX_TARGET=file:https:///index stellar/lighthorizon-index-batch

# Push images
- if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/lighthorizon'
Expand Down
2 changes: 1 addition & 1 deletion exp/lighthorizon/index/cmd/map.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ for (( i=0; i < $BATCH_COUNT; i++ ))
do
echo -n "Creating map job $i... "

NETWORK_PASSPHRASE='testnet' MODULES='accounts_unbacked,transactions' \
NETWORK_PASSPHRASE='testnet' JOB_INDEX_ENV='AWS_BATCH_JOB_ARRAY_INDEX' MODULES='accounts_unbacked,transactions' \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here as on the other PR - why do we need the name of the name of the value rather than just passing the value itself? e.g. JOB_INDEX=$AWS_BATCH_JOB_ARRAY_INDEX?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, good point, this would be a good simplification, might squeeze it in.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I think I see why, under different orchestration environments like k8s, aws, that layer injects the job index number as a container env variable, the name can be anything, AWS_BATCH_JOB_ARRAY_INDEX or JOB_COMPLETION_INDEX for k8s, and there is no shell layer like here for interpolation, the only way the app code which got launched in the container deployed on orchestration layer can realistically reference/find the value is to be given the known deployment env var key name for job index instead and app code then performs the 2nd look up against that in env space to get the value.

AWS_BATCH_JOB_ARRAY_INDEX=$i BATCH_SIZE=$BATCH_SIZE FIRST_CHECKPOINT=$FIRST \
TXMETA_SOURCE=file:https://$1 INDEX_TARGET=file:https://$2 WORKER_COUNT=1 \
./map &
Expand Down
2 changes: 1 addition & 1 deletion exp/lighthorizon/index/cmd/reduce.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ for (( i=0; i < $REDUCE_JOB_COUNT; i++ ))
do
echo -n "Creating reduce job $i... "

AWS_BATCH_JOB_ARRAY_INDEX=$i MAP_JOB_COUNT=$MAP_JOB_COUNT \
AWS_BATCH_JOB_ARRAY_INDEX=$i JOB_INDEX_ENV="AWS_BATCH_JOB_ARRAY_INDEX" MAP_JOB_COUNT=$MAP_JOB_COUNT \
REDUCE_JOB_COUNT=$REDUCE_JOB_COUNT WORKER_COUNT=4 \
INDEX_SOURCE_ROOT=file:https://$1 INDEX_TARGET=file:https://$2 \
timeout -k 30s 10s ./reduce &
Expand Down