A load balancer / proxy / gateway for presto compute engine.
Step 1: setup mysql. Install docker and run the below command when setting up first time:
docker run -d -p 3306:3306 --name mysqldb -e MYSQL_ROOT_PASSWORD=root123 -e MYSQL_DATABASE=prestogateway -d mysql:5.7
Next time onwards, run the following commands to start mysqldb
docker start mysqldb
Now open mysql console and install the presto-gateway tables:
mysql -uroot -proot123 -h127.0.0.1 -Dprestogateway
Once logged in to mysql console, please run gateway-ha-persistence.sql to populate the tables.
Step 2: Edit the configuration gateway-ha-config.yml
Step 3: Add below program argument to class HaGatewayLauncher and debug in IDE
server /path/to/gateway-ha/src/test/resources/config-template.yml
run mvn clean install to build presto-gateway
Edit the config file and update the mysql db information.
cd gateway-ha/target/
java -jar gateway-ha-{{VERSION}}-jar-with-dependencies.jar server ../gateway-ha-config.yml
Now you can access load balanced presto at localhost:8080 port. We will refer to this as prestogateway.lyft.com
curl -X POST http:https://localhost:8080/entity?entityType=GATEWAY_BACKEND \
-d '{ "name": "presto1", \
"proxyTo": "http:https://presto1.lyft.com",\
"active": true, \
"routingGroup": "adhoc" \
}'
curl -X POST http:https://localhost:8080/entity?entityType=GATEWAY_BACKEND \
-d '{ "name": "presto2", \
"proxyTo": "http:https://presto2.lyft.com",\
"active": true, \
"routingGroup": "adhoc" \
}'
If the backend URL is different from the proxyTo URL (for example if they are internal vs. external hostnames). You can use the optional externalUrl field to override the link in the Active Backends page.
curl -X POST http:https://localhost:8080/entity?entityType=GATEWAY_BACKEND \
-d '{ "name": "presto1", \
"proxyTo": "http:https://presto1.lyft.com",\
"active": true, \
"routingGroup": "adhoc" \
"externalUrl": "http:https://presto1-external.lyft.com",\
}'
curl -X POST http:https://localhost:8080/entity?entityType=GATEWAY_BACKEND \
-d '{ "name": "presto2", \
"proxyTo": "http:https://presto2.lyft.com",\
"active": true, \
"routingGroup": "adhoc" \
"externalUrl": "http:https://presto2-external.lyft.com",\
}'
curl -X GET http:https://localhost:8080/entity/GATEWAY_BACKEND
[
{
"active": true,
"name": "presto1",
"proxyTo": "http:https://presto1.lyft.com",
"routingGroup": "adhoc"
},
{
"active": true,
"name": "presto2",
"proxyTo": "http:https://presto2.lyft.com",
"routingGroup": "adhoc"
}
]
curl -X POST -d "presto3" http:https://localhost:8080/gateway/backend/modify/delete
curl -X POST http:https://localhost:8080/gateway/backend/deactivate/presto2
curl -X GET http:https://localhost:8080/gateway/backend/active | python -m json.tool
[{
"active": true,
"name": "presto1",
"proxyTo": "http:https://presto1.lyft.com",
"routingGroup": "adhoc"
}]
curl -X POST http:https://localhost:8080/gateway/backend/activate/presto2
PrestoGateway records history of recent queries and displays links to check query details page in respective presto cluster.
The Gateway admin page is used to configure the gateway to multiple backends. Existing backend information can also be modified using the same.
For resource group and selector apis, we can now specify a query parameter with the request supporting multiple presto databases for different presto backends. This allows a user to configure a db for every presto backend with their own resource groups and selector tables. To use this, just specify the query parameter ?useSchema= to the request. Example, to list all resource groups,
curl -X GET http:https://localhost:8080/presto/resourcegroup/read/{INSERT_ID_HERE}?useSchema=newdatabasename
To add a single resource group, specify all relevant fields in the body.
curl -X POST http:https://localhost:8080/presto/resourcegroup/create \
-d '{ "resourceGroupId": 1, \
"name": "resourcegroup1", \
"softMemoryLimit": "100%", \
"maxQueued": 100, \
"softConcurrencyLimit": 100, \
"hardConcurrencyLimit": 100, \
"environment": "test", \
"schedulingPolicy": null, \
"schedulingWeight": null, \
"jmxExport": null, \
"softCpuLimit": null, \
"hardCpuLimit": null, \
"parent": null, \
"environment": "test" \
}'
If no resourceGroupId (type long) is specified, then all existing resource groups are fetched.
curl -X GET http:https://localhost:8080/presto/resourcegroup/read/{INSERT_ID_HERE}
Specify all columns in the body, which will overwrite properties for the resource group with that specific resourceGroupId.
curl -X POST http:https://localhost:8080/presto/resourcegroup/update \
-d '{ "resourceGroupId": 1, \
"name": "resourcegroup_updated", \
"softMemoryLimit": "80%", \
"maxQueued": 50, \
"softConcurrencyLimit": 40, \
"hardConcurrencyLimit": 60, \
"environment": "test", \
"schedulingPolicy": null, \
"schedulingWeight": null, \
"jmxExport": null, \
"softCpuLimit": null, \
"hardCpuLimit": null, \
"parent": null, \
"environment": "test" \
}'
To delete a resource group, specify the corresponding resourceGroupId (type long).
curl -X POST http:https://localhost:8080/presto/resourcegroup/delete/{INSERT_ID_HERE}
To add a single selector, specify all relevant fields in the body.
curl -X POST http:https://localhost:8080/presto/selector/create \
-d '{ "resourceGroupId": 1, \
"priority": 1, \
"userRegex": "selector1", \
"sourceRegex": "resourcegroup1", \
"queryType": "insert" \
}'
If no resourceGroupId (type long) is specified, then all existing selectors are fetched.
curl -X GET http:https://localhost:8080/presto/selector/read/{INSERT_ID_HERE}
To update a selector, the existing selector must be specified with all relevant fields under "current". The updated version of that selector is specified under "update", with all relevant fields included. If the selector under "current" does not exist, a new selector will be created with the details under "update". Both "current" and "update" must be included to update a selector.
curl -X POST http:https://localhost:8080/presto/selector/update \
-d '{ "current": {
"resourceGroupId": 1, \
"priority": 1, \
"userRegex": "selector1", \
"sourceRegex": "resourcegroup1", \
"queryType": "insert" \
},
"update": {
"resourceGroupId": 1, \
"priority": 2, \
"userRegex": "selector1_updated", \
"sourceRegex": "resourcegroup1", \
"queryType": null \
}
}'
To delete a selector, specify all relevant fields in the body.
curl -X POST http:https://localhost:8080/presto/selector/delete \
-d '{ "resourceGroupId": 1, \
"priority": 2, \
"userRegex": "selector1_updated", \
"sourceRegex": "resourcegroup1", \
"queryType": null \
}'
To add a single global property, specify all relevant fields in the body.
curl -X POST http:https://localhost:8080/presto/globalproperty/create \
-d '{
"name": "cpu_quota_period", \
"value": "1h" \
}'
If no name (type String) is specified, then all existing global properties are fetched.
curl -X GET http:https://localhost:8080/presto/globalproperty/read/{INSERT_NAME_HERE}
Specify all columns in the body, which will overwrite properties for the global property with that specific name.
curl -X POST http:https://localhost:8080/presto/globalproperty/update \
-d '{
"name": "cpu_quota_period", \
"value": "2h" \
}'
To delete a global property, specify the corresponding name (type String).
curl -X POST http:https://localhost:8080/presto/globalproperty/delete/{INSERT_NAME_HERE}
Presto gateway supports graceful shutdown of Presto clusters. Even when a cluster is deactivated, any submitted query states can still be retrieved based on the Query ID.
To graceful shutdown a Presto cluster without query losses, the steps are:
- Set the backend to deactivate state, this prevents any new incoming queries from getting assigned to the backend.
- Poll the Presto backend coorinator URL until the queued query count and the running query count both hit 0.
- Terminate the Presto Coordinator & Worker Java process.
To gracefully shutdown a single worker process, see this for the operations.
Want to help build Presto Gateway? Check out our contributing documentation