Skip to content

A write-buffer implementation in django that temporarily stores write requests in-memory before writing them on disk.

Notifications You must be signed in to change notification settings

OmairK/write-buffer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Write Buffer

Implementation details

This API will be tested at peak times of the day when there are write requests incoming, so the primary objective here was to ease the load database as the Postgresql server would be the first bottleneck here. For that a redis write-buffer is implemented that will store the validated data into redis, Redis’s Reliable Queue pattern. A celery scheduler will then pop <user_defined_value> of elements from redis and write it onto the Postgresql database via a bulk insert operation, which will be much faster than a single writes hence reducing the load on the database.

Settings up

  • Pre-requisites
    • docker-compose
docker-compose build
docker-compose up
  • Endpoint Open API spec Redoc link

  • Benchmarking After running the docker container a benchmark test can be carried out via

cd misc/test-data
python benchmark.py

About

A write-buffer implementation in django that temporarily stores write requests in-memory before writing them on disk.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published