Make the Redis Mass Insertion by using the REdis Serialization Protocol (RESP) simple.
pip install resppython \
-m resp \ # the installed module 'resp'
-i /path/to/dump.txt \ # the path to the input file
-d ',' \ # the delimiter (default: ',')
-r 'RPUSH {0} {1} | SET {0} {2}' \ # the Redis command(s)
| redis-cli --pipeIf the order of the data isn't relevant, you can split the origin file in multiple chunks and pipe them to the script concurrently:
cat /path/to/dump.txt | parallel -j4 --block 1M -I ,, --pipe 'python -m resp -d "," -r "SET {0} {1}" -p' | redis-cli --pipeIf parallel isn't already installed on your system, you can install it with sudo apt-get install parallel on Linux or brew install parallel on macOS.
python -m resp [-h] -r REDIS [-i INPUT] [-d DELIMITER] [-p]
python -m resp [--help] --redis REDIS [--input INPUT] [--delimiter DELIMITER] [--pipe]Install the required environment modules by executing the script environment.sh:
conda env create -n resp python=2 -f environment.yml
source activate resppython -m unittest discover -vp '*Test.py'Don't be shy and feel free to contact me by e-mail or on Twitter.
The module is Open Source Software released under the MIT license.