RFC - Embed Docker/Machine

[bradrydzewski]

I would like to create a new type of plugin for automatically adding servers (workers) to Drone via the user interface. We should use the docker/machine project as much as possible, since it already integrates with a bunch of providers, and handles installing and configuring Docker and generating SSL keys for secure communication over TCP.

I'm not 100% what this would look like, but here are some thoughts:

[amazon]
access_key = ""
secret_key = ""
session_token = ""
ami = ""
region = ""
vpc_id = ""
subnet_id = ""
zone = ""
security_group = ""
instance_type = ""
root_size = ""

The above toml description pretty much mirrors the flags defined for the amazon module in Docker machine. See the following:
https://github.com/docker/machine/blob/master/drivers/amazonec2/amazonec2.go#L75

We could then create a plugin in Drone at /plugin/hosting/amazon, initialize the plugin similar to GitHub and others, and this plugin could import the docker/machine/amazon package directly to create, start and stop servers.

The interface for a Host could look like this:

func Create() (Server, error)
func Destroy(Server) error

We should register each server with the worker pool on creation, and unregister when destroyed. We should persist the list of servers in the Drone database. We should register all servers with the worker pool when Drone first starts (by fetching the list from the database).

It would be great if we could start with a Digital Ocean implementation as a proof of concept to get things started.

[bradrydzewski]

Alternate proposal to investigate docker swarm. I need to research this a bit further to understand the implications.

[kapilt]

swarm already works afaics, it just publishes the docker api, just drop it in it as a worker, and repeat for n workers. machine use case is a little different for dynamic creation of workers.

[bradrydzewski]

@kapilt we need to support the affinity parameters so that service containers (ie mysql, postgres, etc) are executed on the same server as your build container. We would use a combination of swarm and machine to manage a cluster of drone servers as opposed to configuring them in the toml file, since it gives much more flexibility.

[shawnzhu]

@bradrydzewski it requires parameter -e affinity:container==<first-service-container-name> with command docker run to do so. See docker swarm affinity filter.

My thought is, instead of introducing the idea local mode and swarm node for running drone, it would be simpler to use affinity parameter as default when creating service/build container in current design.

I also saw #906 where it will support docker compose when orchestrate service containers (awesome!), where docker compose supports affinity environment as well when running with swarm. So I would say affinity parameter is the default option for drone's container orchestration.

I'd like to contribute a PR to add this feature if it makes sense to you.

[bradrydzewski]

the reason we need two modes (local mode and swarm mode) is because swarm manages the allocation of server resources. This will essentially manage how many builds Drone can run at once. When not using Swarm, Drone will need an internal mechanism to determine how many builds it can run concurrently.

[gtaylor]

I'd be looking to be able to dynamically specify kubernetes nodes, too, FWIW. I'm not familiar enough with swarm to know whether it would cover this particular usage case.

[bradrydzewski]

added support for docker-machine via the new command line utility:
http:https://readme.drone.io/cli/machines.html