Intro

Memory Sharing Multiprocessing Tool for Parallelizing Function Apply on Large Python Objects (i.e., np.array / pd.DataFrame)

TODO:

Allow sharing of numpy array between processes forked by os.fork
Allow sharing of numpy array between processes created by billiard
Allow sharing of Pandas DataFrame between processes created by billiard
- SharedDataFrame class implemented (pd_share.py)
- Use SharedDataFrame with billiard
Explain the code of this repo in README. (installation / run)
Build multi-thread version parallelism code using rust language
Shared Memory for AutoML (parallel computation)
- Make sure sharedmem work with sklearn models (in grid_search)
- Make sure sharedmem work with all models (sklearn + xgboost + lightgbm) (see ml_models.py)
- Make sure SharedDataFrame work with xgboost!
- Build my own version of parallel grid search!
Try experimenting with Ray tune
- Run tutorial
- Study the usage
- Fix compatibility

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
.ipynb_checkpoints		.ipynb_checkpoints
__pycache__		__pycache__
.gitignore		.gitignore
Book_Purchased.csv		Book_Purchased.csv
LICENSE		LICENSE
Playground.ipynb		Playground.ipynb
README.md		README.md
billiard_with_shared_np_array.py		billiard_with_shared_np_array.py
billiard_with_shared_pd.py		billiard_with_shared_pd.py
forked_processes_with_shared_np_array.py		forked_processes_with_shared_np_array.py
grid_search.py		grid_search.py
ml_models.py		ml_models.py
pd_share.py		pd_share.py
ray_tune.py		ray_tune.py
requirements.txt		requirements.txt