-
Notifications
You must be signed in to change notification settings - Fork 5
/
distributed-otp-applications.html
469 lines (351 loc) · 27.9 KB
/
distributed-otp-applications.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="https://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="Content-Style-Type" content="text/css" />
<meta name="keywords" content="Erlang, distributed OTP, application, takeover, failover, distributed application, dist_ac, node, cluster, restart" />
<meta name="description" content="A shorter chapter on distributed OTP applications, showing how to use OTP mechanisms to allow failover/takeover mechanisms to make systems redundant and reliable." />
<meta name="google-site-verification" content="mi1UCmFD_2pMLt2jsYHzi_0b6Go9xja8TGllOSoQPVU" />
<link rel="stylesheet" type="text/css" href="static/css/screen.css" media="screen" />
<link rel="stylesheet" type="text/css" href="static/css/sh/shCore.css" media="screen" />
<link rel="stylesheet" type="text/css" href="static/css/sh/shThemeLYSE2.css" media="screen" />
<link rel="stylesheet" type="text/css" href="static/css/print.css" media="print" />
<link href="rss" type="application/rss+xml" rel="alternate" title="LYSE news" />
<link rel="icon" type="image/png" href="favicon.ico" />
<link rel="apple-touch-icon" href="static/img/touch-icon-iphone.png" />
<link rel="apple-touch-icon" sizes="72x72" href="static/img/touch-icon-ipad.png" />
<link rel="apple-touch-icon" sizes="114x114" href="static/img/touch-icon-iphone4.png" />
<title>Distributed OTP Applications | Learn You Some Erlang for Great Good!</title>
</head>
<body>
<div id="wrapper">
<div id="header">
<h1>Learn you some Erlang</h1>
<span>for great good!</span>
</div> <!-- header -->
<div id="menu">
<ul>
<li><a href="content.html" title="Home">Home</a></li>
<li><a href="faq.html" title="Frequently Asked Questions">FAQ</a></li>
<li><a href="rss" title="Latest News">RSS</a></li>
<li><a href="static/erlang/learn-you-some-erlang.zip" title="Source Code">Code</a></li>
</ul>
</div><!-- menu -->
<div id="content">
<div class="noscript"><noscript>Hey there, it appears your Javascript is disabled. That's fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!</noscript></div>
<h2>Distributed OTP Applications</h2>
<p>Although Erlang leaves us with a lot of work to do, it still provided a few solutions. One of these is the concept of <em>distributed OTP applications</em>. Distributed OTP applications, or just <em>distributed applications</em> when in the context of OTP, allow to define <em>takeover</em> and <em>failover</em> mechanisms. We'll see what that means, how that works, and write a little demo app to go with it.</p>
<h3><a class="section" name="adding-more-to-otp">Adding More to OTP</a></h3>
<p>If you recall the chapter on OTP applications, we briefly saw the structure of an application as something using a central application controller, dispatching to application masters, each monitoring a top-level supervisor for an application:</p>
<img class="center explanation" src="static/img/application-controller.png" width="593" height="246" alt="The Application controller stands over three application masters (in this graphic, in real life it has many more), which each stand on top of a supervisor process" />
<p>In standard OTP applications, the application can be loaded, started, stopped or unloaded. In distributed applications, we change how things work; now the application controller shares its work with the <em><a class="docs" href="https://www.erlang.org/doc/design_principles/distributed_applications.html">distributed application controller</a></em>, another process sitting next to it (usually called <em>dist_ac</em>):</p>
<img class="center explanation" src="static/img/dist_ac.png" width="432" height="268" alt="The application controller supervises two master which in turn supervise two supervisors. In parallel to the application cantroller is the dist_ac, also supervising its own application" />
<p>Depending on the application file, the ownership of the application will change. A dist_ac will be started on all nodes, and all dist_acs will communicate together. What they talk about is not too relevant, except for one thing. As mentioned earlier, the four application statuses were being loaded, started, stopped, and unloaded; distributed applications split the idea of a started application into <em>started</em> and <em>running</em>.</p>
<p>The difference between both is that you could define an application to be global within a cluster. An application of this kind can only run on one node at a time, while regular OTP applications don't care about whatever's happening on other nodes.</p>
<p>As such a distributed application will be started on all nodes of a cluster, but only running on one.</p>
<p>What does this mean for the nodes where the application is started without being run? The only thing they do is wait for the node of the running application to die. This means that when the node that runs the app dies, another node starts running it instead. This can avoid interruption of services by moving around different subsystems.</p>
<p>Let's see what this means in more detail.</p>
<h3><a class="section" name="taking-and-failing-over">Taking and Failing Over</a></h3>
<p>There are two important concepts handled by distributed applications. The first one is the idea of a <em>failover</em>. A failover is the idea described above of restarting an application somewhere else than where it stopped running.</p>
<p>This is a particularly valid strategy when you have redundant hardware. You run something on a 'main' computer or server, and if it fails, you move it to a backup one. In larger scale deployments, you might instead have 50 servers running your software (all at maybe 60-70% load) and expect the running ones to absorb the load of the failing ones. The concept of failing over is mostly important in the former case, and somewhat least interesting in the latter one. </p>
<p>The second important concept of distributed OTP applications is the <em>takeover</em>. Taking over is the act of a dead node coming back from the dead, being known to be more important than the backup nodes (maybe it has better hardware), and deciding to run the application again. This is usually done by gracefully terminating the backup application and starting the main one instead.</p>
<div class="note">
<p><strong>Note:</strong> In terms of distributed programming fallacies, distributed OTP applications assume that when there is a failure, it is likely due to a hardware failure, and not a netsplit. If you deem netsplits more likely than hardware failures, then you have to be aware of the possibility that the application is running both as a backup and main one, and that funny things could happen when the network issue is resolved. Maybe distributed OTP applications aren't the right mechanism for you in these cases.</p>
</div>
<p>Let's imagine that we have a system with three nodes, where only the first one is running a given application:</p>
<img class="center explanation" src="static/img/failover1.png" width="285" height="108" alt="Three nodes, A, B and C, and the application runs under A" />
<p>The nodes <code>B</code> and <code>C</code> are declared to be backup nodes in case <code>A</code> dies, which we pretend just happened:</p>
<img class="center explanation" src="static/img/failover2.png" width="285" height="100" alt="Two nodes, B and C, and no application" />
<p>For a brief moment, there's nothing running. After a while, <code>B</code> realizes this and decides to take over the application:</p>
<img class="center explanation" src="static/img/failover3.png" width="285" height="114" alt="Two nodes, B and C, and the application runs under B" />
<p>That's a failover. Then, if <code>B</code> dies, the application gets restarted on <code>C</code>:</p>
<img class="center explanation" src="static/img/failover4.png" width="285" height="118" alt="Node C with an application running under it" />
<p>Another failover and all is well and good. Now, suppose that <code>A</code> comes back up. <code>C</code> is now running the app happily, but <code>A</code> is the node we defined to be the main one. This is when a takeover occurs: the app is willingly shut down on <code>C</code> and restarted on <code>A</code>:</p>
<img class="center explanation" src="static/img/failover5.png" width="285" height="108" alt="Two nodes, A andbC, and the application runs under A" />
<p>And so on for all other failures.</p>
<p>One obvious problem you can see is how terminating applications all the time like that is likely to be losing important state. Sadly, that's your problem. You'll have to think of places where to put and share all that vital state away before things break down. The OTP mechanism for distributed applications makes no special case for that.</p>
<p>Anyway, let's move on to see how we could practically make things work.</p>
<h3><a class="section" name="the-magic-8-ball">The Magic 8-Ball</a></h3>
<p>A magic 8-ball is a simple toy that you shake randomly in order to get divine and helpful answers. You ask questions like "Will my favorite sports team win the game tonight?" and the ball you shake replies something like "Without a doubt"; you can then safely bet your house's value on the final score. Other questions like "Should I make careful investments for the future" could return "That is unlikely" or "I'm not sure". The magic 8-ball has been vital in the western world's political decision making in the last decades and it is only normal we use it as an example for fault-tolerance.</p>
<p>Our implementation won't make use of real-life switching mechanisms used to automatically find servers such as DNS round-robins or load balancers. We'll rather stay within pure Erlang and have three nodes (denoted below as <code>A</code>, <code>B</code>, and <code>C</code>) part of a distributed OTP application. The <code>A</code> node will represent the main node running the magic 8-ball server, and the <code>B</code> and <code>C</code> nodes will be the backup nodes:</p>
<img class="center explanation" src="static/img/main-back.png" width="163" height="118" alt="three nodes, A, B, and C, connected together" />
<p>Whenever <code>A</code> fails, the 8-ball application should be restarted on either <code>B</code> or <code>C</code>, and both nodes will still be able to use it transparently.</p>
<p>Before setting things up for distributed OTP applications, we'll first build the application itself. It's going to be mind bogglingly naive in its design:</p>
<img class="center explanation" src="static/img/sup-serv.png" width="170" height="153" alt="A supervisor supervising a server" />
<p>And in total we'll have 3 modules: the supervisor, the server, and the application callback module to start things. The supervisor will be rather trivial. We'll call it <code><a class="source" href="static/erlang/m8ball/src/m8ball_sup.erl">m8ball_sup</a></code> (as in <em>Magic 8 Ball Supervisor</em>) and we'll put it in the <code>src/</code> directory of a standard OTP application:</p>
<pre class="brush:erl">
-module(m8ball_sup).
-behaviour(supervisor).
-export([start_link/0, init/1]).
start_link() ->
supervisor:start_link({global,?MODULE}, ?MODULE, []).
init([]) ->
{ok, {{one_for_one, 1, 10},
[{m8ball,
{m8ball_server, start_link, []},
permanent,
5000,
worker,
[m8ball_server]
}]}}.
</pre>
<p>This is a supervisor that will start a single server (<code><a class="source" href="static/erlang/m8ball/src/m8ball_server.erl">m8ball_server</a></code>), a permanent worker process. It's allowed one failure every 10 seconds.</p>
<p>The magic 8-ball server will be a little bit more complex. We'll build it as a gen_server with the following interface:</p>
<pre class="brush:erl">
-module(m8ball_server).
-behaviour(gen_server).
-export([start_link/0, stop/0, ask/1]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
code_change/3, terminate/2]).
%%%%%%%%%%%%%%%%%
%%% INTERFACE %%%
%%%%%%%%%%%%%%%%%
start_link() ->
gen_server:start_link({global, ?MODULE}, ?MODULE, [], []).
stop() ->
gen_server:call({global, ?MODULE}, stop).
ask(_Question) -> % the question doesn't matter!
gen_server:call({global, ?MODULE}, question).
</pre>
<p>Notice how the server is started using <code>{global, ?MODULE}</code> as a name and how it's accessed with the same tuple for each call. That's the <code><a class="docs" href="https://erldocs.com/17.3/kernel/global.html">global</a></code> module we've seen in the last chapter, applied to behaviours.</p>
<p>Next come the callbacks, the real implementation. Before I show how we build it, I'll mention how I want it to work. The magic 8-ball should randomly pick one of many possible replies from some configuration file. I want a configuration file because it should be easy to add or remove answers as we wish.</p>
<p>First of all, if we want to do things randomly, we'll need to set up some randomness as part of our init function:</p>
<pre class="brush:erl">
%%%%%%%%%%%%%%%%%
%%% CALLBACKS %%%
%%%%%%%%%%%%%%%%%
init([]) ->
<<A:32, B:32, C:32>> = crypto:rand_bytes(12),
random:seed(A,B,C),
{ok, []}.
</pre>
<p>We've seen that pattern before in the <a class="chapter" href="buckets-of-sockets.html">Sockets chapter</a>: we're using 12 random bytes to set up the initial random seed to be used with the <code><a class="docs" href="https://erldocs.com/17.3/stdlib/random.html#uniform/1">random:uniform/1</a></code> function.</p>
<p>The next step is to read the answers from the configuration file and pick one. If you recall the <a class="chapter" href="building-otp-applications.html">OTP application chapter</a>, the easiest way to set up some configuration is to use the <code>app</code> file to do it (in the <code>env</code> tuple). Here's how we're gonna do this:</p>
<pre class="brush:erl">
handle_call(question, _From, State) ->
{ok, Answers} = application:get_env(m8ball, answers),
Answer = element(random:uniform(tuple_size(Answers)), Answers),
{reply, Answer, State};
handle_call(stop, _From, State) ->
{stop, normal, ok, State};
handle_call(_Call, _From, State) ->
{noreply, State}.
</pre>
<p>The first clause shows what we want to do. I expect to have a tuple with all the possible answers within the <code>answers</code> value of the <code>env</code> tuple. Why a tuple? Simply because accessing elements of a tuple is a constant time operation while obtaining it from a list is linear (and thus takes longer on larger lists). We then send the answer back.</p>
<div class="note">
<p><strong>Note:</strong> the server reads the answers with <code>application:get_env(m8ball, answers)</code> on each question asked. If you were to set new answers with a call like <code>application:set_env(m8ball, answers, {"yes","no","maybe"})</code>, the three answers would instantly be the possible choices for future calls.</p>
<p>Reading them once at startup should be somewhat more efficient in the long run, but it will mean that the only way to update the possible answers is to restart the application.</p>
</div>
<p>You should have noticed by now that we don't actually care about the question asked — it's not even passed to the server. Because we're returning random answers, it is entirely useless to copy it from process to process. We're just saving work by ignoring it entirely. We still leave the answer there because it will make the final interface feel more natural. We could also trick our magic 8-ball to always return the same answer for the same question if we felt like it, but we won't bother with that for now.</p>
<p>The rest of the module is pretty much the same as usual for a generic gen_server doing nothing:</p>
<pre class="brush:erl">
handle_cast(_Cast, State) ->
{noreply, State}.
handle_info(_Info, State) ->
{noreply, State}.
code_change(_OldVsn, State, _Extra) ->
{ok, State}.
terminate(_Reason, _State) ->
ok.
</pre>
<p>Now we can get to the more serious stuff, namely the application file and the callback module. We'll begin with the latter, <code><a class="source" href="static/erlang/m8ball/src/m8ball.erl">m8ball.erl</a></code>:</p>
<pre class="brush:erl">
-module(m8ball).
-behaviour(application).
-export([start/2, stop/1]).
-export([ask/1]).
%%%%%%%%%%%%%%%%%
%%% CALLBACKS %%%
%%%%%%%%%%%%%%%%%
start(normal, []) ->
m8ball_sup:start_link().
stop(_State) ->
ok.
%%%%%%%%%%%%%%%%%
%%% INTERFACE %%%
%%%%%%%%%%%%%%%%%
ask(Question) ->
m8ball_server:ask(Question).
</pre>
<p>That was easy. Here's the associated <code>.app</code> file, <code><a class="source" href="static/erlang/m8ball/ebin/m8ball.app">m8ball.app</a></code>:</p>
<pre class="brush:erl">
{application, m8ball,
[{vsn, "1.0.0"},
{description, "Answer vital questions"},
{modules, [m8ball, m8ball_sup, m8ball_server]},
{applications, [stdlib, kernel, crypto]},
{registered, [m8ball, m8ball_sup, m8ball_server]},
{mod, {m8ball, []}},
{env, [
{answers, {<<"Yes">>, <<"No">>, <<"Doubtful">>,
<<"I don't like your tone">>, <<"Of course">>,
<<"Of course not">>, <<"*backs away slowly and runs away*">>}}
]}
]}.
</pre>
<p>We depend on <code>stdlib</code> and <code>kernel</code>, like all OTP applications, and also on <code>crypto</code> for our random seeds in the server. Note how the answers are all in a tuple: that matches the tuples required in the server. In this case, the answers are all binaries, but the string format doesn't really matter — a list would work as well.</p>
<h3><a class="section" name="making-the-application-distributed">Making the Application Distributed</a></h3>
<p>So far, everything was like a perfectly normal OTP application. We have very few changes to add to our files to make it work for a distributed OTP application; in fact, only one function clause to add, back in the <code>m8ball.erl</code> module:</p>
<pre class="brush:erl">
%%%%%%%%%%%%%%%%%
%%% CALLBACKS %%%
%%%%%%%%%%%%%%%%%
start(normal, []) ->
m8ball_sup:start_link();
start({takeover, _OtherNode}, []) ->
m8ball_sup:start_link().
</pre>
<p>The <code>{takeover, OtherNode}</code> argument is passed to <code>start/2</code> when a more important node takes over a backup node. In the case of the magic 8-ball app, it doesn't really change anything and we can just start the supervisor all the same.</p>
<p>Recompile your code and it's pretty much ready. But hold on, how do we define what nodes are the main ones and which ones are backups? The answer is in configuration files. Because we want a system with three nodes (<code>a</code>, <code>b</code>, and <code>c</code>), we'll need three configuration files (I named them <a class="source" href="static/erlang/m8ball/config/a.config">a.config</a>, <a class="source" href="static/erlang/m8ball/config/b.config">b.config</a>, and <a class="source" href="static/erlang/m8ball/config/c.config">c.config</a>, then put them all in <code>config/</code> inside the application directory):</p>
<pre class="brush:erl">
[{kernel,
[{distributed, [{m8ball,
5000,
[a@ferdmbp, {b@ferdmbp, c@ferdmbp}]}]},
{sync_nodes_mandatory, [b@ferdmbp, c@ferdmbp]},
{sync_nodes_timeout, 30000}
]}].
</pre>
<pre class="brush:erl">
[{kernel,
[{distributed, [{m8ball,
5000,
[a@ferdmbp, {b@ferdmbp, c@ferdmbp}]}]},
{sync_nodes_mandatory, [a@ferdmbp, c@ferdmbp]},
{sync_nodes_timeout, 30000}
]}].
</pre>
<pre class="brush:erl">
[{kernel,
[{distributed, [{m8ball,
5000,
[a@ferdmbp, {b@ferdmbp, c@ferdmbp}]}]},
{sync_nodes_mandatory, [a@ferdmbp, b@ferdmbp]},
{sync_nodes_timeout, 30000}
]}].
</pre>
<p>Don't forget to rename the nodes to fit your own host. Otherwise the general structure is always the same:</p>
<pre class="brush:erl">
[{kernel,
[{distributed, [{AppName,
TimeOutBeforeRestart,
NodeList}]},
{sync_nodes_mandatory, NecessaryNodes},
{sync_nodes_optional, OptionalNodes},
{sync_nodes_timeout, MaxTime}
]}].
</pre>
<p>The <var>NodeList</var> value can usually take a form like <code>[A, B, C, D]</code> for <var>A</var> to be the main one, <var>B</var> being the first backup, and <var>C</var> being the next one, and so on. Another syntax is possible, giving a list of like <code>[A, {B, C}, D]</code>, so <var>A</var> is still the main node, <var>B</var> and <var>C</var> are equal secondary backups, then the other ones, etc.</p>
<img class="right" src="static/img/magic-8-ball.png" width="358" height="272" alt="A magic 8-ball showing 'I don't think so'" />
<p>The <code>sync_nodes_mandatory</code> tuple will work in conjunction with <code>sync_nodes_timeout</code>. When you start a distributed virtual machine with values set for this, it will stay locked up until all the mandatory nodes are also up and locked. Then they get synchronized and things start going. If it takes more than <var>MaxTime</var> to get all the nodes up, then they will all crash before starting.</p>
<p>There are way more options available, and I recommend looking into the <a class="docs" href="https://www.erlang.org/doc/man/kernel_app.html">kernel application documentation</a> if you want to know more about them.</p>
<p>We'll try things with the <code>m8ball</code> application now. If you're not sure 30 seconds is enough to boot all three VMs, you can increase the <code>sync_nodes_timeout</code> as you wish. Then, start three VMs:</p>
<pre class="brush:eshell">
$ erl -sname a -config config/a -pa ebin/
</pre>
<pre class="brush:eshell">
$ erl -sname b -config config/b -pa ebin/
</pre>
<pre class="brush:eshell">
$ erl -sname c -config config/c -pa ebin/
</pre>
<p>As you start the third VM, they should all unlock at once. Go into each of the three virtual machines, and turn by turn, start both <code>crypto</code> and <code>m8ball</code> with <code>application:start(AppName)</code>.</p>
<p>You should then be able to call the magic 8-ball from any of the connected nodes:</p>
<pre class="brush:eshell">
(a@ferdmbp)3> m8ball:ask("If I crash, will I have a second life?").
<<"I don't like your tone">>
(a@ferdmbp)4> m8ball:ask("If I crash, will I have a second life, please?").
<<"Of Course">>
</pre>
<pre class="brush:erl">
(c@ferdmbp)3> m8ball:ask("Am I ever gonna be good at Erlang?").
<<"Doubtful">>
</pre>
<p>How motivational. To see how things are, call <code>application:which_applications()</code> on all nodes. Only node <code>a</code> should be running it:</p>
<pre class="brush:erl">
(b@ferdmbp)3> application:which_applications().
[{crypto,"CRYPTO version 2","2.1"},
{stdlib,"ERTS CXC 138 10","1.18"},
{kernel,"ERTS CXC 138 10","2.15"}]
</pre>
<pre class="brush:erl">
(a@ferdmbp)5> application:which_applications().
[{m8ball,"Answer vital questions","1.0.0"},
{crypto,"CRYPTO version 2","2.1"},
{stdlib,"ERTS CXC 138 10","1.18"},
{kernel,"ERTS CXC 138 10","2.15"}]
</pre>
<p>The <code>c</code> node should show the same thing as the <code>b</code> node in that case. Now if you kill the <code>a</code> node (just ungracefully close the window that holds the Erlang shell), the application should obviously no longer be running there. Let's see where it is instead:</p>
<pre class="brush:erl">
(c@ferdmbp)4> application:which_applications().
[{crypto,"CRYPTO version 2","2.1"},
{stdlib,"ERTS CXC 138 10","1.18"},
{kernel,"ERTS CXC 138 10","2.15"}]
(c@ferdmbp)5> m8ball:ask("where are you?!").
<<"I don't like your tone">>
</pre>
<p>That's expected, as <code>b</code> is higher in the priorities. After 5 seconds (we set the timeout to 5000 milliseconds), <code>b</code> should be showing the application as running:</p>
<pre class="brush:erl">
(b@ferdmbp)4> application:which_applications().
[{m8ball,"Answer vital questions","1.0.0"},
{crypto,"CRYPTO version 2","2.1"},
{stdlib,"ERTS CXC 138 10","1.18"},
{kernel,"ERTS CXC 138 10","2.15"}]
</pre>
<p>It runs fine, still. Now kill <code>b</code> in the same barbaric manner that you used to get rid of <code>a</code>, and <code>c</code> should be running the application after 5 seconds:</p>
<pre class="brush:erl">
(c@ferdmbp)6> application:which_applications().
[{m8ball,"Answer vital questions","1.0.0"},
{crypto,"CRYPTO version 2","2.1"},
{stdlib,"ERTS CXC 138 10","1.18"},
{kernel,"ERTS CXC 138 10","2.15"}]
</pre>
<p>If you restart the node <code>a</code> with the same command we had before, it will hang. The config file specifies we need <code>b</code> back for <code>a</code> to work. If you can't expect nodes to all be up that way, you'll need to make maybe <code>b</code> or <code>c</code> optional, for example. So if we start both <code>a</code> and <code>b</code>, then the application should automatically come back, right?</p>
<pre class="brush:erl">
(a@ferdmbp)4> application:which_applications().
[{crypto,"CRYPTO version 2","2.1"},
{stdlib,"ERTS CXC 138 10","1.18"},
{kernel,"ERTS CXC 138 10","2.15"}]
(a@ferdmbp)5> m8ball:ask("is the app gonna move here?").
<<"Of course not">>
</pre>
<p>Aw, shucks. The thing is, for the mechanism to work, the application needs to be started <em>as part of the boot procedure of the node</em>. You could, for instance, start <code>a</code> that way for things to work:</p>
<pre class="brush:erl">
erl -sname a -config config/a -pa ebin -eval 'application:start(crypto), application:start(m8ball)'
...
(a@ferdmbp)1> application:which_applications().
[{m8ball,"Answer vital questions","1.0.0"},
{crypto,"CRYPTO version 2","2.1"},
{stdlib,"ERTS CXC 138 10","1.18"},
{kernel,"ERTS CXC 138 10","2.15"}]
</pre>
<p>And from <code>c</code>'s side:</p>
<pre class="brush:erl">
=INFO REPORT==== 8-Jan-2012::19:24:27 ===
application: m8ball
exited: stopped
type: temporary
</pre>
<p>That's because the <code>-eval</code> option gets evaluated as part of the boot procedure of the VM. Obviously, a cleaner way to do it would be to use releases to set things up right, but the example would be pretty cumbersome if it had to combine everything we had seen before.</p>
<p>Just remember that in general, distributed OTP applications work best when working with releases that ensure that all the relevant parts of the system are in place.</p>
<p>As I mentioned earlier, in the case of many applications (the magic 8-ball included), it's sometimes simpler to just have many instances running at once and synchronizing data rather than forcing an application to run only at a single place. It's also simpler to scale it once that design has been picked. If you need some failover/takeover mechanism, distributed OTP applications might be just what you need.</p>
<ul class="navigation">
<li><a href="distribunomicon.html" title="Previous chapter">< Previous</a></li>
<li><a href="contents.html" title="Index">Index</a></li>
<li><a href="common-test-for-uncommon-tests.html" title="Next chapter">Next ></a></li>
</ul>
</div><!-- content -->
<div id="footer">
<a href="https://creativecommons.org/licenses/by-nc-nd/3.0/" title="Creative Commons License Details"><img src="static/img/cc.png" width="88" height="31" alt="Creative Commons Attribution Non-Commercial No Derivative License" /></a>
<p>Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution Non-Commercial No Derivative License</p>
</div> <!-- footer -->
</div> <!-- wrapper -->
<div id="grass" />
<script type="text/javascript" src="static/js/shCore.js"></script>
<script type="text/javascript" src="static/js/shBrushErlang2.js%3F11"></script>
<script type="text/javascript">
SyntaxHighlighter.defaults.gutter = false;
SyntaxHighlighter.all();
</script>
</body>
</html>