forked from beanstalkd/beanstalkd
-
Notifications
You must be signed in to change notification settings - Fork 1
/
protocol.txt
356 lines (225 loc) · 12 KB
/
protocol.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
= Beanstalk Protocol =
Protocol
--------
The beanstalk protocol runs over TCP using ASCII encoding. Clients connect,
send commands and data, wait for responses, and close the connection. For each
connection, the server processes commands serially in the order in which they
were received and sends responses in the same order. All integers in the
protocol are formatted in decimal and (unless otherwise indicated)
nonnegative.
If a client violates the protocol (such as by sending a request that is not
well-formed or a command that does not exist) or if the server has a serious
error that prevents it from continuing service to the current client, the
server will close the connection.
There is no command to close the connection -- the client may simply close the
TCP connection when it no longer has use for the server. However, beanstalkd
performs very well with a large number of open connections, so it is usually
better for the client to keep its connection open and reuse it as much as
possible. This also avoids the overhead of establishing new TCP connections.
The beanstalk protocol contains two kinds of data: text lines and
unstructured chunks of data. Text lines are used for client commands and server
responses. Chunks are used to transfer job bodies and stats information. Each
job body is an opaque sequence of bytes. The server never inspects or modifies
a job body and always sends it back in its original form. It is up to the
clients to agree on a meaningful interpretation of job bodies.
Job Lifecycle
-------------
A job in beanstalk gets created by a client with the "put" command. During its
life it can be in one of four states: "ready", "reserved", "delayed", or
"buried". After the put command, a job typically starts out ready. It waits in
the ready queue until a worker comes along and runs the "reserve" command. If
this job is next in the queue, it will be reserved for the worker. The worker
will execute the job; when it is finished the worker will send a "delete"
command to delete the job.
Here is a picture of the typical job lifecycle:
put reserve delete
-----> [READY] ---------> [RESERVED] --------> *poof*
Here is a picture with more possibilities:
put with delay release with delay
----------------> [DELAYED] <------------.
| |
| (time passes) |
| |
put v reserve | delete
-----------------> [READY] ---------> [RESERVED] --------> *poof*
^ ^ | |
| \ release | |
| `-------------' |
| |
| kick |
| |
| bury |
[BURIED] <---------------'
|
| delete
`--------> *poof*
Producer Command
----------------
The "put" command is for any process that wants to insert a job into the queue.
It comprises a command line followed by the job body:
put <pri> <delay> <ttr> <bytes>\r\n
<data>\r\n
It inserts a job into the queue.
- <pri> is an integer < 2**32. Jobs with smaller priority values will be
scheduled before jobs with larger priorities. The most urgent priority is 0;
the least urgent priority is 4294967295.
- <delay> is an integer number of seconds to wait before putting the job in
the ready queue. The job will be in the "delayed" state during this time.
- <ttr> -- time to run -- is an integer number of seconds to allow a worker
to run this job. This time is counted from the moment a worker reserves
this job. If the worker does not delete, release, or bury the job within
<ttr> seconds, the job will time out and the server will release the job.
- <bytes> is an integer (currently must be < 2**16) indicating the size of
the job body, not including the trailing "\r\n".
- <data> is the job body -- a sequence of bytes of length <bytes> from the
previous line.
After sending the command line and body, the client waits for a reply, which
may be:
- "INSERTED <id>\r\n" to indicate success.
- <id> is the integer id of the new job
- "BURIED\r\n" if the priority queue data structure is full.
Worker Commands
---------------
A process that wants to consume jobs from the queue uses "reserve", "delete",
"release", and "bury". The first worker command, "reserve", looks like this:
reserve\r\n
This will return a newly-reserved job. If no job is available to be reserved,
beanstalkd will wait to send a response until one becomes available. Once a job
is reserved for the client, the client has limited time to run the job before
the job times out, when the server will put the job back into the ready queue.
The time available can be found by asking the server for the job's stats.
There is only one possible response in the form of a text line followed by the
job body:
RESERVED <id> <pri> <bytes>\r\n
<data>\r\n
- <id> is the job id -- an integer unique to this job in this instance of
beanstalkd.
- <pri> is the priority value set by the put, release, or bury commands.
- <bytes> is an integer indicating the size of the job body, not including
the trailing "\r\n".
- <data> is the job body -- a sequence of bytes of length <bytes> from the
previous line. This is a verbatim copy of the bytes that were originally
sent to the server in the put command for this job.
The delete command removes a job from the server entirely. It is normally used
by the client when the job has successfully run to completion. A client can
only delete jobs that it has reserved or jobs that are buried. The delete
command looks like this:
delete <id>\r\n
- <id> is the job id to delete.
The client then waits for one line of response, which may be:
- "DELETED\r\n" to indicate success.
- "NOT_FOUND\r\n" if the job does not exist or is not either reserved by the
client or buried. This could happen if the job timed out before the client
sent the delete command.
The release command puts a reserved job back into the ready queue (and marks
its state as "ready") to be run by any client. It is normally used when the job
fails because of a transitory error. It looks like this:
release <id> <pri> <delay>\r\n
- <id> is the job id to release.
- <pri> is a new priority to assign to the job.
- <delay> is an integer number of seconds to wait before putting the job in
the ready queue. The job will be in the "delayed" state during this time.
The client expects one line of response, which may be:
- "RELEASED\r\n" to indicate success.
- "BURIED\r\n" if the priority queue data structure is full.
- "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.
The bury command puts a job into the "buried" state. Buried jobs are put into a
FIFO linked list and will not be touched by the server again until a client
kicks them with the "kick" command.
The bury command looks like this:
bury <id> <pri>\r\n
- <id> is the job id to release.
- <pri> is a new priority to assign to the job.
There are two possible responses:
- "BURIED\r\n" to indicate success.
- "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.
Other Commands
--------------
The peek command lets the client inspect a job in the system. Its format:
peek[ <id>]\r\n
If <id> is not given, peek will show the next job in the list of buried jobs,
if any.
- <id> is the job id to show, if given.
There are two possible responses, either a single line:
- "NOT_FOUND\r\n" if the requested job doesn't exist or there are no buried
jobs.
Or a line followed by a chunk of data, if the command was successful:
FOUND <id> <pri> <bytes>\r\n
<data>\r\n
- <id> is the job id.
- <pri> is the job's priority.
- <bytes> is an integer indicating the size of the job body, not including
the trailing "\r\n".
- <data> is the job body -- a sequence of bytes of length <bytes> from the
previous line.
The kick command moves jobs into the ready queue. If there are any buried jobs,
it will only kick buried jobs. Otherwise it will kick delayed jobs. It looks
like:
kick <bound>\r\n
- <bound> is an integer upper bound on the number of jobs to kick. The server
will kick no more than <bound> jobs.
The response is of the form
KICKED <count>\r\n
- <count> is an integer indicating the number of jobs actually kicked.
The stats command gives statistical information. If a job id is given, the
response will contain information about that job if it exists. Otherwise stats
will return information about the system as a whole. Its form is:
stats[ <id>]\r\n
If a job id is given, the response may be:
- "NOT_FOUND\r\n" if the job does not exist.
Otherwise the server will respond:
OK <bytes>\r\n
<data>\r\n
- <bytes> is the size of the following data section in bytes.
- <data> is a sequence of bytes of length <bytes> from the previous line. It
is a YAML file with statistical information represented a dictionary.
Stats format
The stats data is a YAML file representing a single dictionary of strings to
scalars.
- The stats dict for a job contains these keys:
- "id" is the job id
- "state" is "ready" or "delayed" or "reserved" or "buried"
- "age" is the time in seconds since the put command that created this job.
- "time-left" is the number of seconds left until the server puts this job
into the ready queue. This number is only meaningful if the job is
reserved or delayed. If the job is reserved and this amount of time
elapses before its state changes, it is considered to have timed out.
- "timeouts" is the number of times this job has timed out during a
reservation.
- "releases" is the number of times a client has released this job from a
reservation.
- "buries" is the number of times this job has been buried.
- "kicks" is the number of times this job has been kicked.
- The stats dict for the system contains these keys:
- "current-jobs-urgent" is the number of ready jobs with priority < 1024.
- "current-jobs-ready" is the number of jobs in the ready queue.
- "current-jobs-reserved" is the number of jobs reserved by all clients.
- "current-jobs-delayed" is the number of delayed jobs.
- "current-jobs-buried" is the number of buried jobs.
- "limit-max-jobs-ready" is the total capacity of the ready queue. Once this
many jobs are ready, any additional jobs will be buried.
- "cmd-put" is the cumulative number of put commands.
- "cmd-peek" is the cumulative number of peek commands.
- "cmd-reserve" is the cumulative number of reserve commands.
- "cmd-delete" is the cumulative number of delete commands.
- "cmd-release" is the cumulative number of release commands.
- "cmd-bury" is the cumulative number of bury commands.
- "cmd-kick" is the cumulative number of kick commands.
- "cmd-stats" is the cumulative number of stats commands.
- "job-timeouts" is the cumulative count of times a job has timed out.
- "total-jobs" is the cumulative count of jobs created.
- "current-connections" is the number of currently open connections.
- "current-producers" is the number of open connections that have each
issued at least one put command.
- "current-workers" is the number of open connections that have each issued
at least one reserve command.
- "current-waiting" is the number of open connections that have issued a
reserve command but not yet received a response.
- "total-connections" is the cumulative count of connections.
- "pid" is the process id of the server.
- "version" is the version string of the server.
- "rusage-utime" is the accumulated user CPU time of this process in seconds
and microseconds.
- "rusage-stime" is the accumulated system CPU time of this process in
seconds and microseconds.
- "uptime" is the number of seconds since this server started running.