Web based metrics, monitoring, and observer.

We are talking about :wobserver at ElixirConf 2017. Check out the presentation and samples and our other talk about Task Bunny.

Click to view images

Functionality:

• Drop-in monitoring though web interface.
• Metrics endpoint (/metrics) for system monitoring. (Default: Prometheus)
• Monitoring automation through JSON API.
• Node management and discovery behind firewalls and load balancers.
• Easy to extend:
  • Add custom metrics and pages for your project, just by adding them in the config.
  • Just 3 lines of code to add pages/metrics for your library, when users have :wobserver installed. (See how.)

• wobserver
• Table of contents
• Installation
  • Hex
  • Build manually
• Usage
  • Browser
  • API
    • Remote nodes
    • System
    • Allocators
    • Application
    • Process
    • Ports
    • Table view
  • Metrics
• Configuration
  • Port
  • Mode
    • Standalone
    • Plug
  • Node Discovery
  • Metrics
    • Add Metrics
    • Formatting
  • Pages
    • Config
    • Dynamically
• Library Integration
• Improvements
• Contributors
• license

Add Wobserver as a dependency to your mix.exs file:

def deps do
  [{:wobserver, "~> 0.1"}]
end

and add it to your list of applications:

def application do
  [applications: [:wobserver]]
end

Then run mix deps.get in your shell to fetch the dependencies.

Note: Check out plug mode to integrate with a Phoenix or other web application. (Prevents startup of separate web server.)

Run the following commands to build the project:

$ npm install
$ mix deps.get
$ mix build

Note: Use the package generated by mix build if you want to include the local wobserver in your application. (Unpack in your deps.)

Wobserver does not support being included directly from github. The required assets are not included in the repo in build form and can therefore not be used. It is possible to build locally and use the generated package. (See Build manually for more information.)

To view the web interface just enter https://<host>[:<port>]/ in the browser and it should show the :wobserver interface. The default port is 4001, but the port can be changed in the configuration.

A sample interface can be viewed here.

The API can be accessed by calling https://<host>[:<port>]/api/. The index will return 404, but specific endpoints should return results.

The API provides a list of remote nodes by calling https://<host>[:<port>]/api/nodes.

The API of remote nodes can be accessed by calling the API endpoint and prefixing the node name, host, or host:port.

For example considering the following node list:

[
  {
    "port": 4001,
    "name": "node_prime",
    "local?": true,
    "host": "192.168.5.55"
  },
  {
    "port": 80,
    "name": "remote",
    "local?": false,
    "host": "80.23.1.165"
  }
]

The following calls would all work for the first node: (local is a reserved name that always points to the local node.)

https://<host>[:<port>]/api/local/system
https://<host>[:<port>]/api/node_prime/system
https://<host>[:<port>]/api/192.168.5.55/system
https://<host>[:<port>]/api/192.168.5.55:4001/system

And these calls would work for the second node:

https://<host>[:<port>]/api/remote/system
https://<host>[:<port>]/api/80.23.1.165/system
https://<host>[:<port>]/api/80.23.1.165:80/system

The API provides a list of system information by calling https://<host>[:<port>]/api/system.

The scheduler is a list of load values (0-1) for each scheduler.

Example:

{
  "statistics": {
    "uptime": 459876,
    "process_total": 122,
    "process_running": 0,
    "process_max": 262144,
    "output": 1259201,
    "input": 12945380
  },
  "scheduler": [
    0.0037370416873916392,
    0.0003088661849770247,
    0.0003072993680801981,
    0.00030274231847091137,
    0.0004706952361156354,
    0.00028556537348788645,
    0.00025471141618606366,
    0.0002522242536713918
  ],
  "memory": {
    "total": 30275576,
    "process": 5242800,
    "ets": 886544,
    "code": 13635797,
    "binary": 288744,
    "atom": 594561
  },
  "cpu": {
    "schedulers_online": 8,
    "schedulers_available": 8,
    "schedulers": 8,
    "logical_processors_online": 8,
    "logical_processors_available": "unknown",
    "logical_processors": 8
  },
  "architecture": {
    "wordsize_internal": 8,
    "wordsize_external": 8,
    "threads": true,
    "thread_pool_size": 10,
    "system_architecture": "x86_64-apple-darwin15.6.0",
    "smp_support": true,
    "otp_release": "19",
    "kernel_poll": false,
    "erts_version": "8.2",
    "elixir_version": "1.4.0"
  }
}

The API provides a list of allocators and their size by calling https://<host>[:<port>]/api/allocators.

Example:

[
  {
    "type": "sl_alloc",
    "carrier": 294912,
    "block": 664
  },
  {
    "type": "std_alloc",
    "carrier": 1081344,
    "block": 498184
  },
  {
    "type": "ll_alloc",
    "carrier": 35913728,
    "block": 26080144
  },
  {
    "type": "eheap_alloc",
    "carrier": 9830400,
    "block": 2634720
  },
  {
    "type": "ets_alloc",
    "carrier": 3178496,
    "block": 890880
  },
  ...
]

The API provides a list of applications and their descriptions by calling https://<host>[:<port>]/api/application.

The information for a specific application, including the process hierarchy can be found by calling https://<host>[:<port>]/api/application/<application-name>.

Example: https://localhost:4001/api/application

[
  {
    "version": "0.1.0",
    "name": "wobserver",
    "description": "Web based metrics, monitoring, and observer."
  },
  {
    "version": "1.3.0",
    "name": "plug",
    "description": "A specification and conveniences for composable modules between web applications"
  },
  {
    "version": "1.1.0",
    "name": "cowboy",
    "description": "Small, fast, modular HTTP server."
  },
  {
    "version": "1.2.1",
    "name": "ranch",
    "description": "Socket acceptor pool for TCP protocols."
  },
  {
    "version": "0.6.1",
    "name": "credo",
    "description": "A static code analysis tool for the Elixir language with a focus on code consistency and teaching."
  },
  {
    "version": "0.2.0",
    "name": "bunt",
    "description": "256 color ANSI coloring in the terminal"
  },
  {
    "version": "1.6.5",
    "name": "hackney",
    "description": "simple HTTP client"
  },
  {
    "version": "1.4.0",
    "name": "logger",
    "description": "logger"
  },
  ...
]

https://localhost:4001/api/application/elixir

{
  "pid": "#PID<0.59.0>",
  "name": "<0.59.0>",
  "meta": {
    "status": "waiting",
    "init": "proc_lib.init_p/5",
    "current": "application_master.main_loop/2",
    "class": "application"
  },
  "children": [
    {
      "pid": "#PID<0.60.0>",
      "name": "<0.60.0>",
      "meta": {
        "status": "waiting",
        "init": "application_master.start_it/4",
        "current": "application_master.loop_it/4",
        "class": "unknown"
      },
      "children": [
          {
            "pid": "#PID<0.61.0>",
            "name": "elixir_sup",
            "meta": {
              "status": "waiting",
              "init": "proc_lib.init_p/5",
              "current": "gen_server.loop/6",
              "class": "supervisor"
            },
            "children": [
              {
              "pid": "#PID<0.62.0>",
              "name": "elixir_config",
              "meta": {
                "status": "waiting",
                "init": "proc_lib.init_p/5",
                "current": "gen_server.loop/6",
                "class": "gen_server"
              },
              "children": []
            },
            {
              "pid": "#PID<0.63.0>",
              "name": "elixir_code_server",
              "meta": {
                "status": "waiting",
                "init": "proc_lib.init_p/5",
                "current": "gen_server.loop/6",
                "class": "gen_server"
              },
              "children": []
            }
          ]
        }
      ]
    }
  ]
}

The API provides a list of processes and their basic information by calling https://<host>[:<port>]/api/process.

The information for a specific process, including a links, memory usage, and state can be found by calling https://<host>[:<port>]/api/application/<process-name>.

The process name can be given as pid, name, or short pid.

So all the following are valid:

https://localhost:4001/api/process/<0.247.0>
https://localhost:4001/api/process/#PID<0.247.0>   # Rememeber to url encode # -> %23
https://localhost:4001/api/process/Wobserver.Supervisor

Example: https://localhost:4001/api/process

{
  "processes": [
    {
      "reductions": 162714,
      "pid": "#PID<0.247.0>",
      "nr1": "0",
      "message_queue_length": 0,
      "memory": 11888,
      "init": "timer_server",
      "current": "gen_server.loop/6"
    },
    {
      "reductions": 95,
      "pid": "#PID<0.243.0>",
      "nr1": "0",
      "message_queue_length": 0,
      "memory": 2792,
      "init": "erlang.apply/2",
      "current": "io.execute_request/2"
    },
    {
      "reductions": 954,
      "pid": "#PID<0.242.0>",
      "nr1": "0",
      "message_queue_length": 0,
      "memory": 16808,
      "init": "Elixir.IEx.Evaluator.init/4",
      "current": "Elixir.IEx.Evaluator.loop/3"
    },
    ...
  ]
}

https://localhost:4001/api/process/<0.247.0>

{
  "trap_exit": true,
  "state": "[]",
  "relations": {
    "links": [
      "#PID<0.53.0>"
    ],
    "group_leader": "#PID<0.33.0>",
    "ancestors": [
      "kernel_safe_sup",
      "kernel_sup",
      "#PID<0.34.0>"
    ]
  },
  "registered_name": "timer_server",
  "priority": "normal",
  "pid": "#PID<0.247.0>",
  "meta": {
    "status": "waiting",
    "init": "proc_lib.init_p/5",
    "current": "gen_server.loop/6",
    "class": "gen_server"
  },
  "message_queue_len": 0,
  "memory": {
    "total": 0,
    "stack_size": 9,
    "stack_and_heap": 1974,
    "heap_size": 1598,
    "gc_min_heap_size": 233,
    "gc_full_sweep_after": 65535
  },
  "error_handler": "error_handler"
}

The API provides a list of ports and their owners by calling https://<host>[:<port>]/api/ports.

Example: https://localhost:4001/api/ports

[
  {
    "output": 0,
    "os_pid": "undefined",
    "name": "forker",
    "links": [],
    "input": 0,
    "id": 0,
    "connected": "#PID<0.0.0>"
  },
  {
    "output": 3,
    "os_pid": "undefined",
    "name": "efile",
    "links": [
      "#PID<0.4.0>"
    ],
    "input": 46,
    "id": 8,
    "connected": "#PID<0.4.0>"
  },
  {
    "output": 18810,
    "os_pid": "undefined",
    "name": "efile",
    "links": [
      "#PID<0.44.0>"
    ],
    "input": 23874,
    "id": 4680,
    "connected": "#PID<0.44.0>"
  },
  ...
]

The API provides a list of tables and their details by calling https://<host>[:<port>]/api/table.

The information for a specific details, including a the actual data can be found by calling https://<host>[:<port>]/api/table/<table-name>.

Example: https://localhost:4001/api/table Example:

[
  {
    "type": "set",
    "size": 0,
    "protection": "protected",
    "owner": "#PID<0.247.0>",
    "name": "timer_interval_tab",
    "meta": {
      "write_concurrency": false,
      "read_concurrency": false,
      "compressed": false
    },
    "memory": 304,
    "id": "timer_interval_tab"
  },
  {
    "type": "ordered_set",
    "size": 7,
    "protection": "protected",
    "owner": "#PID<0.247.0>",
    "name": "timer_tab",
    "meta": {
      "write_concurrency": false,
      "read_concurrency": false,
      "compressed": false
    },
    "memory": 304,
    "id": "timer_tab"
  },
  {
    "type": "set",
    "size": 7,
    "protection": "public",
    "owner": "#PID<0.228.0>",
    "name": "workstore",
    "meta": {
      "write_concurrency": false,
      "read_concurrency": false,
      "compressed": false
    },
    "memory": 5138,
    "id": 417840
  },
  ...
]

https://localhost:4001/api/table/timer_interval_tab

{
  "type": "set",
  "size": 0,
  "protection": "protected",
  "owner": "#PID<0.247.0>",
  "name": "timer_interval_tab",
  "meta": {
    "write_concurrency": false,
    "read_concurrency": false,
    "compressed": false
  },
  "memory": 304,
  "id": "timer_interval_tab",
  "data": []
}

Metrics are available by calling https://<host>[:<port>]/metrics.

The metrics are by default formatted for Prometheus, but can be configured to work with any system. An explanation of how to configure the metrics format and how to add metrics to the output will be added later.

https://localhost:4001/metrics

# HELP erlang_vm_used_memory_bytes Memory usage of the Erlang VM.
# TYPE erlang_vm_used_memory_bytes gauge
erlang_vm_used_memory_bytes{node="10.74.181.35",type="atom"} 553593
erlang_vm_used_memory_bytes{node="10.74.181.35",type="binary"} 359552
erlang_vm_used_memory_bytes{node="10.74.181.35",type="code"} 13533686
erlang_vm_used_memory_bytes{node="10.74.181.35",type="ets"} 1899472
erlang_vm_used_memory_bytes{node="10.74.181.35",type="process"} 6048552
# HELP erlang_vm_used_io_bytes IO counter for the Erlang VM.
# TYPE erlang_vm_used_io_bytes counter
erlang_vm_used_io_bytes{node="10.74.181.35",type="input"} 11301316
erlang_vm_used_io_bytes{node="10.74.181.35",type="output"} 618157

The port can be set in the config by setting :port for :wobserver to a valid number.

config :wobserver,
  port: 80

Wobserver runs by default in :standalone mode. This means that :wobserver will start its own :cowboy listeners on a separate port. Standalone mode is ideal for drop-in web viewing, but might not be ideal if another part of the application is already running an web server. It is possible to enable :plug mode to prevent :wobserver from starting :cowboy to handle web requests.

Standalone mode is the default operating mode. A :cowboy (ranch) listener will be started with 10 accepters and a websocket. Set mode to :standalone in the :wobserver configuration to force standalone mode.

Example:

config :wobserver,
  mode: :standalone

Plug mode prevents :wobserver from starting :cowboy (ranch). Set mode to :plug in the :wobserver configuration to use plug mode. Set remote_url_prefix to the url prefix you put :wobserver behind to make sure dns node discovery still functions.

Plug mode prevents :wobserver from starting :cowboy (ranch). Set mode to :plug in the :wobserver configuration to use plug mode. Set remote_url_prefix to the url prefix you put :wobserver behind to make sure dns node discovery still functions.

Add the following line of code to the application's router to forward requests to :wobserver:

  forward "/wobserver", to: Wobserver.Web.Router

Add the following option to the :cowboy child_spec to enable use of the :wobserver websocket:

dispatch: [
    {:_, [
      {"/ws", Wobserver.Web.Client, []},
      {:_, Cowboy.Handler, {<your own router>, []}}
    ]}
  ],

Add the following line of code to the Phoenix router to forward requests to :wobserver:

  forward "/wobserver", Wobserver.Web.Router

Add the following option to your Phoenix applications Endpoint to enable use of the :wobserver websocket (the path should match what is in the 'forward' in your router):

  socket "/wobserver", Wobserver.Web.PhoenixSocket

config.exs

config :wobserver,
  mode: :plug,
  remote_url_prefix: "/wobserver"

my_router.ex

defmodule MyApp.MyRouter do
  use Plug.Router

  plug :match
  plug :dispatch

  forward "/wobserver", to: Wobserver.Web.Router
end

application.ex

defmodule MyApp.Application do
  use Application

  alias Plug.Adapters.Cowboy

  def start(_type, _args) do
    import Supervisor.Spec, warn: false

    options = [
      dispatch: [
        {:_, [
          {"/wobserver/ws", Wobserver.Web.Client, []},
          {:_, Cowboy.Handler, {MyApp.MyRouter, []}}
        ]}
      ],
    ]

    children = [
      Cowboy.child_spec(:http, MyApp.MyRouter, [], options)
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

The method used can be set in the config file by setting:

config :wobserver,
  discovery: :none

The following methods can be used: (default: :none)

• :none, just returns the local node.
• :dns, use DNS to search for other nodes. The option discovery_search needs to be set to filter entries.
• :custom, a function as String.

No discovery: (default)

config :wobserver,
  port: 80,
  discovery: :none

Using dns as discovery service:

config :wobserver,
  port: 80,
  discovery: :dns,
  discovery_search: "google.nl"

Using a custom function:

config :wobserver,
  port: 80,
  discovery: :custom,
  discovery_search: &MyApp.CustomDiscovery.discover/0

Using an anonymous function:

config :wobserver,
  port: 80,
  discovery: :custom,
  discovery_search: fn -> [] end

Both the custom and anonymous functions can be given as a String, which will get evaluated.

Metrics and metric generators can be added by setting them in the configuration.

To add custom metrics set the :metrics option. The :metrics option must be a keyword list with the following keys:

• additional, for a keyword list with additional metrics.
• generators, for a list of metric generators.

The following settings are accepted for additional:

• keyword list, the key is the name of the metric and the value is the metric data.

The following inputs are accepted for metric generators:

• list of callable functions. Every function should return a keyword list with as key the name of the metric and as value the metric data.

For more information about how to format metric data see: Wobserver.Util.Metrics.Formatter.format_all/1.

For example this configuration:

config :wobserver,
  metrics: [
    additional: [
      example: {fn -> [red: 5] end, :gauge, "Description"},
    ],
    generators: [
      "&MyApp.generator/0",
      fn -> [bottles: {fn -> [wall: 8, floor: 10] end, :gauge, "Description"}] end
      fn -> [server: {"MyApp.Server.metrics/0", :gauge, "Description"}] end
    ]
  ]

Metrics and metric generators can also be added dynamically at runtime.

To register a metric you need to pass a keyword list to Wobserver.register with the same data as you would set in the configuration file.

For example:

Wobserver.register :metric, [example: {fn -> [red: 5] end, :gauge, "Description"}]

To register a metric generator you need to pass a list of functions to Wobserver.register.

For example:

Wobserver.register :metric, [&MyLibrary.Metrics.generate/0]

A custom formatter can be created for output of metrics by implementing the Wobserver.Util.Metrics.Formatter behavior. This custom formatter can be enabled in the configuration file by setting metric_format.

For example this configuration:

config :wobserver,
  metric_format: JsonFormatter

And this simple JSON formatter:

defmodule SimpleJsonFormatter do
  @behaviour Wobserver.Util.Metrics.Formatter

  def format_data(name, data, type, help) do
    formatted_data =
      data
      |> Enum.map(fn {value, labels} ->
           %{value: value, labels: Enum.into(labels, %{})}
         end)

    %{
      name: name,
      type: type,
      description: help,
      data: formatted_data
    }
    |> Poison.encode!
  end

  def combine_metrics(metrics) do
    "[" <> Enum.join(metrics,",") <> "]"
  end

  def merge_metrics(metrics) do
    "[" <> Enum.join(metrics,",") <> "]"
  end
end

Produce the following output:

[
  [
    {
      "type": "gauge",
      "name": "erlang_vm_used_memory_bytes",
      "description": "Memory usage of the Erlang VM.",
      "data": [
        {
          "value": 654241,
          "labels": {
            "type": "atom",
            "node": "192.168.1.88"
          }
        },
          {
          "value": 503464,
          "labels": {
            "type": "binary",
            "node": "192.168.1.88"
          }
        },
        {
          "value": 14459399,
          "labels": {
            "type": "code",
            "node": "192.168.1.88"
          }
        },
        {
          "value": 2073072,
          "labels": {
            "type": "ets",
            "node": "192.168.1.88"
          }
        },
        {
          "value": 6008488,
          "labels": {
            "type": "process",
            "node": "192.168.1.88"
          }
        }
      ]
    },
    {
      "type": "counter",
      "name": "erlang_vm_used_io_bytes",
      "description": "IO counter for the Erlang VM.",
      "data": [
        {
          "value": 29523254,
          "labels": {
            "type": "input",
            "node": "192.168.1.88"
          }
        },
        {
          "value": 9960593,
          "labels": {
            "type": "output",
            "node": "192.168.1.88"
          }
        }
      ]
    }
  ]
]

Pages are custom views in the web interface and endpoints in the JSON API for an application or library.

There are two ways to add a custom page:

• config, set a list of custom pages in the mix config.
• registration, call Wobserver.register/2 and dynamically add pages.

Adding more pages to :wobserver can be done by setting the :pages option.

The :pages option must be a list of page data.

The page data can be formatted as:

• {title, command, callback}
• {title, command, callback, options}
• a map with the following fields:
  • title
  • command
  • callback
  • options (optional)

For more information and types see: Wobserver.Page.register/1.

Example:

config :wobserver,
  pages: [
    {"Example", :example, fn -> %{x:  9} end}
  ]

Dynamically register a page with :wobserver by calling Wobserver.register/2.

The following inputs are accepted:

• {title, command, callback}
• {title, command, callback, options}
• a map with the following fields:
  • title
  • command
  • callback
  • options (optional)

The fields are used as followed:

• title, the name of the page. Is used for the web interface menu.
• command, single atom to associate the page with.
• callback, function to be evaluated, when the a api is called or page is viewd. The result is converted to JSON and displayed.
• options, options for the page.

The following options can be set:

• api_only (boolean), if set to true the page won't show up in the web interface, but will only be available as API.
• refresh (float, 0-1), sets the refresh time factor. Used in the web interface to refresh the data on the page. Set to 0 for no refresh.

Example:

Wobserver.register(:page, {"My App", :my_app, fn -> %{data: 123} end})

Integrating a library with :wobserver is done by calling Wobserver.register/2, when the library loads, and dynamically adding pages and metrics.

To safely integrate with :wobserver use the following code:

if Code.ensure_loaded(Wobserver) == {:module, Wobserver} do
  Wobserver.register :page, {"My Library", :my_library, fn -> %{data: 123} end}
  Wobserver.register :metric, [&MyLibrary.Metrics.generate/0]
end

The above code will make sure that the library only calls register, when :wobserver is loaded. This will prevent the library from trying to register, when :wobserver is not installed.

For an implementation see the :task_bunny library: TaskBunny, lib/task_bunny.ex.

The above code will generate warnings while compiling the library.

warning: function Wobserver.register/2 is undefined (module Wobserver is not available)

There are two options to remove those warnings.

The mix.exs file can be edited to excluded Wobserver from reference checks.

To do this add the following line to project/0 in your mix file:

  xref: [exclude: [Wobserver]]

The code can be rewritten to use Kernel.apply/3. The following code will be less readable and slightly slower, but will not generate warnings.

if Code.ensure_loaded(Wobserver) == {:module, Wobserver} do
  apply Wobserver, :register, [:page, {"My Library", :my_library, fn -> %{data: 123} end}]
  apply Wobserver, :register, [:metric, [&MyLibrary.Metrics.generate/0]]
end

• Cleanup namespaces.
• Cleanup readme, condense sample output.
• Overhaul web interface (make fancier/pleasant)

• OvermindDL1 - Phoenix Socket support and lots of issue reports.

Wobserver source code is released under the MIT License. Check LICENSE file for more information.