This is a Golang (go) program to read from a Message Queue System and Store into Database using concurrency as you can see into the image below
- The number of consumers could be different from the numbers of workers
- The process (job) of consume one message from queue and store into the database is synchronous because every message needs to be acknowledge (confirm as storage).
source code
git clone https://github.com/christiangda/mq-to-db.git
cd mq-to-db/
go run -race ./cmd/mq-to-db/main.go --help
# and then
go run -race ./cmd/mq-to-db/main.go --configFile config-sample.yamlNOTE: the parameter -raceis to check race conditions because we are using Go Concurrency
binary
./mq-to-db --helpRabbitMQ
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-managementNOTE:
- RabbitMQ web console: https://localhost:15672
- Username: guest
- Password: guest
PostgreSQL
docker run --rm --name postgresql -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres
# logs
docker logs postgresql -f
# remember to stop and remove (--rm in docker run do it for you)
docker stop postgresqlmq-to-db
go run -race ./cmd/mq-to-db/main.go --configFile config-sample.yamlo
make
./mq-to-db --configFile config-sample.yamlManually
go build \
-o mq-to-db \
-ldflags "-X github.com/christiangda/mq-to-db/internal/version.Version=$(git rev-parse --abbrev-ref HEAD) -X github.com/christiangda/mq-to-db/internal/version.Revision=$(git rev-parse HEAD) -X github.com/christiangda/mq-to-db/internal/version.Branch=$(git rev-parse --abbrev-ref HEAD) -X github.com/christiangda/mq-to-db/internal/version.BuildUser=\"$(git config --get user.name | tr -d '\040\011\012\015\n')\" -X github.com/christiangda/mq-to-db/internal/version.BuildDate=$(date +'%Y-%m-%dT%H:%M:%S')" \
./cmd/mq-to-db/main.goAutomatic
makeNOTES related to make
-
- This create a linux binary (64bits) and also Docker Image (linux 64 bit)
-
- Check the
Makefileto see themake targets
- Check the
- https://www.openmymind.net/The-Little-Go-Book/
- https://golang.org/doc/effective_go.html#generality
- https://golang.org/doc/code.html
- https://golang.org/pkg/database/sql/
- https://golang.org/s/sqldrivers
- https://astaxie.gitbooks.io/build-web-application-with-golang/content/en/05.4.html
- https://gist.github.com/divan/eb11ddc97aab765fb9b093864410fd25
- https://www.alexedwards.net/blog/interfaces-explained
- https://golang.org/doc/effective_go.html#interfaces_and_types
- https://github.com/go-yaml/yaml
- https://www.sohamkamani.com/golang/2018-07-19-golang-omitempty/
- https://ubuntu.com/blog/api-v3-of-the-yaml-package-for-go-is-available
- https://gigi.nullneuron.net/gigilabs/rabbitmq-who-creates-the-queues-and-exchanges/
- https://www.rabbitmq.com/queues.html#optional-arguments
- https://www.rabbitmq.com/dlx.html
- https://www.rabbitmq.com/vhosts.html
- https://www.rabbitmq.com/tutorials/tutorial-one-go.html
- https://www.inanzzz.com/index.php/post/0aeg/creating-a-rabbitmq-producer-example-with-golang
### Metrics
- https://prometheus.io/docs/guides/go-application/
- https://eltonminetto.net/en/post/2020-03-13-golang-prometheus/
- https://scot.coffee/2018/12/monitoring-go-applications-with-prometheus/
- https://dev.to/ahmedash95/understand-golang-channels-and-how-to-monitor-with-grafana-part-2-2-3o42
### Tests
- https://blog.alexellis.io/inject-build-time-vars-golang/
- https://goenning.net/2017/01/25/adding-custom-data-go-binaries-compile-time/
- https://ewencp.org/blog/golang-iterators/
- https://pkg.go.dev/google.golang.org/api/iterator?tab=doc#example-package-ServerPages
- https://github.com/googleapis/google-cloud-go/wiki/Iterator-Guidelines