Building and installing the MQTT plugin #15

mmohamed15 · 2021-08-06T17:24:49Z

i have a small problem in building the Grafana mqtt plugin
i tried to build it in windows but i failed specially when i run the yarn build command i always get that -rm and mage in not recognized
then i switched to try to build it in ubuntu linux but i also faced the problem of mage is not recognized even though i had installed go and tried to install mage
can you help me in setting the necessary setup to be able to build the mqtt plugin correctly i hope that it will be bundled with grafana 8.1 release and this didn't happen
maybe how i can install mage orn something like that

atifali · 2021-08-06T18:08:47Z

make sure to satisfy the pre-requisites from here: https://grafana.com/tutorials/build-a-streaming-data-source-plugin/
for mage follow the instructions here (assuming the system already has golang installed): https://magefile.org/

whbruce · 2021-08-06T19:36:44Z

Here's what I had to do on Ubuntu 20. It's non-trivial...

Installing

Ubuntu 20 supports Grafana v7.0, but we need Grafana 8.0 as it supports streaming data sources (e.g. MQTT).

$ sudo apt-get install -y gnupg2 curl
$ curl https://packages.grafana.com/gpg.key | sudo apt-key add -
$ sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
$ sudo apt-get update
$ sudo apt-get -y install grafana

Dashboard will be served on port 3000. Default username and password are "admin", you'll be prompted to change password.

Add MQTT Data Source

Install prerequisites

Yarn

$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
$ sudo apt update
$ sudo apt -y install yarn

Node v14

$ curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -
$ sudo apt update
$ sudo apt -y install nodejs
$ node --version
v14.17.4

Go

$ sudo apt -y install golang
$ mkdir -p ~/go
$ go version
go version go1.13.8 linux/amd64

Add the following to your environment (e.g. .bashrc)

$ export GOPATH=~/go
$ export PATH=$PATH:$GOPATH/bin

Mage

$ go get -u -d github.com/magefile/mage
$ cd $GOPATH/src/github.com/magefile/mage
$ go run bootstrap.go
$ mage -version
Mage Build Tool v1.11.0-2-g4cf3cfc
Build Date: 2021-08-03T11:57:28-07:00
Commit: 4cf3cfc
built with: go1.13.8

Build data source

$ mkdir grafana-plugins
$ cd grafana-plugins
$ https://github.com/grafana/mqtt-datasource.git
$ cd mqtt-datasource
$ yarn build
$ yarn install

Add data source to grafana

Edit vi /etc/grafana/grafana.ini.

Add data source folder

Look for # Directory where grafana will automatically scan and look for plugins and update as follows

# Directory where grafana will automatically scan and look for plugins
;plugins = /var/lib/grafana/plugins
plugins = /path/to/grafana-plugins

Allow unsigned sources

Look for [plugins] section and update as follows

[plugins]
;enable_alpha = false
;app_tls_skip_verify_insecure = false
# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
allow_loading_unsigned_plugins = grafana-mqtt-datasource

Restart service

sudo systemctl start grafana-server

mmohamed15 · 2021-08-06T19:53:06Z

can you help me for the same steps for windows

whbruce · 2021-08-06T20:05:45Z

Sorry, can't help, I'm only working on Linux. And I'm only a few days ahead of you in ramping on this project.

mmohamed15 · 2021-08-06T20:32:49Z

Ok i found a small hint you need to change the the command rm to rimraf in the package.json file i will try to make a guide to for building this plugin for windows

mmohamed15 · 2021-08-06T21:43:46Z

How to Build the grafana MQTT plugin for windows 10 :-

1.Download Nodejs version 14 or higher from https://nodejs.org/
2.Install Nodejs using recommended settings
3.To install yarn go to windows start menu choose the cmd and type "npm install --global yarn"
4.Download Go language for windows from https://golang.org/
5.Install Go language
6.Download Mage from github repository https://github.com/magefile/mage by clicking the code (green button ) and choose download zip
7.Extract the downloaded zip file to the path of GO lanuguage
8.From Windows start menu open cmd and go the bath of the extracted mage folder
9.Type "go run bootstrap.go"
10.In the mqtt-datasource-plugin open the file package.json and change the written commands in the fifth line from after the build command from "rm" to "rimraf" and save the file
11.Proceed with the written procedure to install the plugin as described in the readme file

atifali · 2021-08-07T17:21:16Z

@whbruce @mmohamed15 appreciate the detailed pre-requisite installation guides 👏
have referenced these in the docs, see: #16. closing this issue now.

DonatoD · 2021-09-16T15:24:21Z

Sorry, it's normal, on window, have all this warning?

KirannBhavaraju · 2021-10-01T12:46:21Z

@whbruce and @mmohamed15 , isnt the npm package @grafana/toolkit found here, installed as npm i @grafana/toolkit -g also a dependency to for the yarn build step to be successful?

It is under devDependencies in the package.json so running yarn install before doing yarn build step should work.

Potemkinkuliss · 2021-10-27T22:18:09Z

Im am trying to install this plugin inside a container with grafana installed running ubuntu (grafana:8.2.2-ubuntu) on a linux mashine.
Everything works until go run bootstrap.go. I receive:

root@fb9bae54db75:~/go/src/github.com/magefile/mage# go build -o /root/go/bin/mage -ldflags=-X "github.com/magefile/mage/mage.timestamp=2021-10-27T22:12:25Z" -X "github.com/magefile/mage/mage.commitHash=4cf3cfc" -X "github.com/magefile/mage/mage.gitTag=v1.11.0-2-g4cf3cfc" github.com/magefile/mage
can't load package: package github.com/magefile/mage/mage.timestamp=2021-10-27T22:12:25Z: malformed module path "github.com/magefile/mage/mage.timestamp=2021-10-27T22:12:25Z": invalid char '='
can't load package: package -X: malformed module path "-X": leading dash
can't load package: package github.com/magefile/mage/mage.commitHash=4cf3cfc: malformed module path "github.com/magefile/mage/mage.commitHash=4cf3cfc": invalid char '='
can't load package: package github.com/magefile/mage/mage.gitTag=v1.11.0-2-g4cf3cfc: malformed module path "github.com/magefile/mage/mage.gitTag=v1.11.0-2-g4cf3cfc": invalid char '='

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"

Any thoughts?

SStarosielec · 2021-11-28T20:21:53Z

10.In the mqtt-datasource-plugin open the file package.json and change the written commands in the fifth line from after the build command from "rm" to "rimraf"

it works better to replace "rm -rf dist && ..." by "del /F /Q dist && ..." on windows.

Anto79-ops · 2022-01-06T15:55:43Z

hi, wondering if you can help. I have Ubuntu 20 and followed these instructions as written above except

ran "yarn install" first than "yarn build" at the end there
No errors in any of the steps, except some dependency warnings like @DonatoD reported

Continuing with the instructions, I modified the config file as described, but still, the plugin is NOT showing up in data source add-on/ pluging search when in Grafana desktop. what have I done wrong? more details here

felice64 · 2022-01-08T16:43:14Z

Hi There,
I did follow all steps recommended by @whbruce and everything wend fine (apparently). I installed it on Ubuntu 20.04
I can see the MQTT plugin in Grafana. When I try to select it I get this error:

Fetch error: 404 Not Found Instantiating http:https://localhost:3000/public/plugins/grafana-mqtt-datasource/module.js Loading plugins/grafana-mqtt-datasource/module
I am not sure how to solve this. Any help is welcomed.
Thanks

Anto79-ops · 2022-01-08T17:07:34Z

I did get this to work, eventually, on my Ubuntu 20 computer (which is a relatively clean install).
This is what I did, followed the above instructions, exactly, except

ran "yarn install" first than "yarn build" at the end there
No errors in any of the steps, except some dependency warnings like @DonatoD reported.
after building, I moved the built plug-in (whole mqtt-datasource folder) to the default grafana directory for plug-ins (you need sudo permissions to move it to this directory)

/var/lib/grafana/plugins

Then, I only changed this in the ini file

[plugins]
;enable_alpha = false
;app_tls_skip_verify_insecure = false
# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
allow_loading_unsigned_plugins = grafana-mqtt-datasource

restarted grafana server, and worked no problem trying to add the mqtt plug-in on the desktop search

My orignal problem was that my own plug-in folder did not have the right permissions, so, placing it into the default grafana plugin fodler solved that.

felice64 · 2022-01-09T05:43:49Z

I did get this to work, eventually, on my Ubuntu 20 computer (which is a relatively clean install). This is what I did, followed the above instructions, exactly, except

ran "yarn install" first than "yarn build" at the end there

No errors in any of the steps, except some dependency warnings like @DonatoD reported.

after building, I moved the built plug-in (whole mqtt-datasource folder) to the default grafana directory for plug-ins (you need sudo permissions to move it to this directory)

/var/lib/grafana/plugins

Then, I only changed this in the ini file
[plugins]
;enable_alpha = false
;app_tls_skip_verify_insecure = false
# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
allow_loading_unsigned_plugins = grafana-mqtt-datasource
restarted grafana server, and worked no problem trying to add the mqtt plug-in on the desktop search

My orignal problem was that my own plug-in folder did not have the right permissions, so, placing it into the default grafana plugin fodler solved that.

Thanks for this confirmation. It is exactly what I did. Step by step, exactly the same.
I resolved the previous issue, however now when I try to save the datasource is says "Plugin unavailable".
Not sure about what to check t this point.

Anto79-ops · 2022-01-09T21:35:08Z

@felice64 I may suggest that you check the Grafana logs, you can change the level of detail in the the config file. Perhaps there's information there you can check out, especilly if you built the plug-in without any errros, and now its just a matter of Grafana recognizing it.

ranimkayali · 2022-05-20T10:36:58Z

I didn't understand the punkt 8th . can you explaine more

mmohamed15 · 2022-05-20T14:10:57Z

Hi ranim
simply you need to open the command line prompt from the windows start menu by opening the search and typing cmd
a black window will be opened that looks like dos operating system
then you need to use the cd command to go to the location of the folder on the hard drive that you extracted Mage in it
for any more help don't hesitate to ask

youduda · 2022-06-09T13:00:55Z

Here is how I manged to build it with the current grafana container:

docker run --rm grafana/grafana:latest -v /path/to/grafana/files:/var/lib/grafana grafana/grafana:8.5.5 exec --user root grafana bash

cd /root/
apk add git go yarn npm
go get github.com/magefile/mage
mkdir grafana-plugins
cd grafana-plugins
git clone https://github.com/grafana/mqtt-datasource.git
cd mqtt-datasource

npm install @grafana/toolkit
PATH="/root/grafana-plugins/mqtt-datasource/node_modules/.bin:/root/go/bin:$PATH"

yarn build
yarn install

chown grafana:root dist/ -R
mv dist mqtt-datasource
mv mqtt-datasource /var/lib/grafana/plugins/

# Now start Grafana with the new plugin
docker run --rm grafana/grafana:latest -p 3000:3000 -v /path/to/grafana/files:/var/lib/grafana -e GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=grafana-mqtt-datasource grafana/grafana:8.5.5

badrbouslikhin · 2022-06-10T12:04:19Z

Based on all the feedback above, here's how I've done it using a Dockerfile to build on my target the plugin:

Clone mqtt-datasource repo

Add the following Dockerfile to the repo root:

FROM node:lts-alpine3.16 as js-builder

WORKDIR /mqtt-datasource

RUN apk add mage --repository=http:https://dl-cdn.alpinelinux.org/alpine/edge/testing

COPY ./ /mqtt-datasource/
RUN yarn install
RUN yarn build

FROM scratch AS export-stage
COPY --from=js-builder mqtt-datasource/dist .

Build using: DOCKER_BUILDKIT=1 docker build --file Dockerfile --output grafana-mqtt-datasource .
Move grafana-mqtt-datasource to grafana/plugins

This was tested on macOS and Linux.

rogerio-pellarin · 2022-12-01T21:12:25Z

Hi guys.
We are almost close to reach the success, but still this datasource is not working...
The following config is the most recent and does everything fine.

docker-compose.yml
`version: '3.8'

services:
grafana:
container_name: grafana
build:
context: .
dockerfile: Dockerfile
user: 0:0
environment:
GF_SECURITY_ADMIN_USER: banana
GF_SECURITY_ADMIN_PASSWORD: grafana
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: grafana-mqtt-datasource
ports:
- "3000:3000"`

Dockerfile
`FROM grafana/grafana:8.5.15-ubuntu

USER root
RUN apt update
RUN apt -y upgrade
RUN apt-get install -y gnupg curl git locate
RUN updatedb

#YARN
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt update && apt -y install yarn

#NODE
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash -
RUN apt -y install nodejs

#GO
WORKDIR /root
RUN curl -OL https://go.dev/dl/go1.19.linux-amd64.tar.gz
RUN tar -xvf go1.19.linux-amd64.tar.gz
RUN mv go /usr/local
RUN mkdir -p /go
RUN export GOPATH=/go
RUN export PATH=$PATH:/usr/local/go/bin
ENV PATH="/usr/local/go/bin:${PATH}"
RUN echo $PATH

#MAGE
RUN git clone https://github.com/magefile/mage
WORKDIR /root/mage
RUN go run bootstrap.go

#GRAFANA PLUGIN
WORKDIR /var/lib/grafana/plugins/
RUN git clone https://github.com/grafana/mqtt-datasource.git
WORKDIR /var/lib/grafana/plugins/mqtt-datasource
RUN yarn
RUN yarn add @grafana/toolkit
ENV PATH="/grafana-plugins/mqtt-datasource/node_modules/.bin:$PATH"
RUN ln -s /root/go/bin/mage /usr/bin/mage
RUN yarn build
RUN chown grafana:root dist/ -R
RUN mv dist mqtt-datasource
RUN mage reloadPlugin
`

The Grafana recognize this plugin, but when we click to add, shows this error.

Anyone has any clue about that?

tomy983 · 2022-12-11T14:37:16Z

[SOLVED]
My bad.. I just followed @whbruce guide, but the requirements are different now. GO minimum version is now 1.16+
It would be great if your answer could be upgraded as it is part of the installation instructions.

Upgraded GO and it now works fine.

OLD QUESTION:

Hi, attempting install of this plugin on Ubuntu 20.04.5 LTS (VM on gcp..

$ yarn -v
1.22.19

node --version
v16.18.1

go version
go version go1.13.8 linux/amd64

mage -version
Mage Build Tool v1.14.0-1-ga920604
Build Date: 2022-12-11T14:01:36Z
Commit: a920604
built with: go1.13.8

Here the output of $ yarn build :

yarn run v1.22.19
$ rm -rf dist && grafana-toolkit plugin:build && mage build:backend
  Using Node.js v16.18.1
  Using @grafana/toolkit v9.1.0
✔ Preparing
⠋ Linting  =============
  
  WARNING: You are currently running a version of TypeScript which is not officially supported by @typescript-eslint/typescript-estree.
  
  You may find that it works just fine, or you may not.
  
  SUPPORTED TYPESCRIPT VERSIONS: >=3.3.1 <4.7.0
  
  YOUR TYPESCRIPT VERSION: 4.7.4
  
  Please only submit bug reports when using the officially supported version.
  
  =============
✔ Linting
 PASS  src/handleEvent.test.ts

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   2 passed, 2 total
Time:        4.995 s
Ran all test suites with tests matching "".
✔ Running tests
⠇ Compiling...  
   assets by path *.md 3.46 KiB
    asset README.md 3.4 KiB [emitted] [from: ../README.md] [copied]
    asset CHANGELOG.md 63 bytes [emitted] [from: ../CHANGELOG.md] [copied]
  asset LICENSE 11.1 KiB [emitted] [from: ../LICENSE] [copied]
  asset module.js 4.88 KiB [emitted] [minimized] (name: module) 1 related asset
  asset img/mqtt.svg 1.31 KiB [emitted] [from: img/mqtt.svg] [copied]
  asset plugin.json 1000 bytes [emitted] [from: plugin.json] [copied]
  orphan modules 7.63 KiB [orphan] 4 modules
  runtime modules 937 bytes 4 modules
  built modules 8.12 KiB [built]
    modules by path external "@grafana/ 126 bytes
      external "@grafana/data" 42 bytes [built] [code generated]
      external "@grafana/runtime" 42 bytes [built] [code generated]
      external "@grafana/ui" 42 bytes [built] [code generated]
    ./module.ts + 4 modules 7.92 KiB [built] [code generated]
    external "react" 42 bytes [built] [code generated]
    external "lodash" 42 bytes [built] [code generated]
  webpack 5.75.0 compiled successfully in 22287 ms 
  
✔ Compiling...
build command-line-arguments: cannot load embed: malformed module path "embed": missing dot in first path element
Error: error compiling magefiles
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Any idea why this embed error?
Thanks

DonatoD · 2022-12-12T09:02:52Z

[SOLVED]
Hi, I'm not able any more in installing mage on windows.
Following this point seems not to work any more:

6.Download Mage from github repository https://github.com/magefile/mage by clicking the code (green button ) and choose download zip
7.Extract the downloaded zip file to the path of GO lanuguage
8.From Windows start menu open cmd and go the bath of the extracted mage folder
9.Type "go run bootstrap.go"

can someone try it?
One more thing. Can you claify which are supposed to be to path of points 7 and 8? I have tryed several way, from within the folder itself and from the path where the main folder is. But always with errors like

Running target: Install
exec: go "env" "GOBIN"
exec: go "env" "GOPATH"
exec: git "rev-parse" "--short" "HEAD"
fatal: not a git repository (or any of the parent directories): .git
exec: git "describe" "--tags"
fatal: not a git repository (or any of the parent directories): .git
exec: go "build" "-o" "C:\\Users\\Itsme\\go\\bin\\mage.exe" "-ldflags=-X \"github.com/magefile/mage/mage.timestamp=2022-12-12T09:43:47+01:00\" -X \"github.com/magefile/mage/mage.commitHash=\" -X \"github.com/magefile/mage/mage.gitTag=dev\"" "github.com/magefile/mage"

or
CreateFile bootstrap.go: The system cannot find the file specified.

[SOLVED]
ok, to make it working easily just follow the instraction https://github.com/magefile/mage without manually downloading and unzipping it. From a cmd windows as administrator do the following:

git clone https://github.com/magefile/mage
cd mage
go run bootstrap.go

DonatoD · 2022-12-12T09:30:22Z

mmmmmm
Now I got this error
Fetch error: 404 Not Found Instantiating http:https://localhost:3000/public/app/plugins/datasource/dist/module Loading app/plugins/datasource/dist/module
Grafana 9.3.1, but I even tried on a fresh 9.2.7 installatin. Always the same error (Windows OS)

[SOLVED]
Ok. You don't need to follow the instraction any more. It has added to the plugin, so it is easier and faster just install it by the plagin section within Grafana

atifali mentioned this issue Aug 7, 2021

include pre-requisite installation details in docs #16

Merged

atifali closed this as completed Aug 7, 2021

whbruce mentioned this issue Aug 22, 2021

Error when trying to yarn build #21

Closed

galigula2 mentioned this issue Jan 28, 2022

Error install plugins #34

Open

ionut-gheorghe mentioned this issue Oct 18, 2022

Query error #51

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building and installing the MQTT plugin #15

Building and installing the MQTT plugin #15

mmohamed15 commented Aug 6, 2021

atifali commented Aug 6, 2021

whbruce commented Aug 6, 2021

mmohamed15 commented Aug 6, 2021

whbruce commented Aug 6, 2021

mmohamed15 commented Aug 6, 2021

mmohamed15 commented Aug 6, 2021 •

edited

Loading

atifali commented Aug 7, 2021

DonatoD commented Sep 16, 2021

KirannBhavaraju commented Oct 1, 2021 •

edited

Loading

Potemkinkuliss commented Oct 27, 2021

SStarosielec commented Nov 28, 2021

Anto79-ops commented Jan 6, 2022 •

edited

Loading

felice64 commented Jan 8, 2022

Anto79-ops commented Jan 8, 2022 •

edited

Loading

felice64 commented Jan 9, 2022

Anto79-ops commented Jan 9, 2022

ranimkayali commented May 20, 2022

mmohamed15 commented May 20, 2022

youduda commented Jun 9, 2022

badrbouslikhin commented Jun 10, 2022 •

edited

Loading

rogerio-pellarin commented Dec 1, 2022

tomy983 commented Dec 11, 2022 •

edited

Loading

DonatoD commented Dec 12, 2022 •

edited

Loading

DonatoD commented Dec 12, 2022 •

edited

Loading

Building and installing the MQTT plugin #15

Building and installing the MQTT plugin #15

Comments

mmohamed15 commented Aug 6, 2021

atifali commented Aug 6, 2021

whbruce commented Aug 6, 2021

Installing

Add MQTT Data Source

Install prerequisites

Yarn

Node v14

Go

Mage

Build data source

Add data source to grafana

Add data source folder

Allow unsigned sources

Restart service

mmohamed15 commented Aug 6, 2021

whbruce commented Aug 6, 2021

mmohamed15 commented Aug 6, 2021

mmohamed15 commented Aug 6, 2021 • edited Loading

atifali commented Aug 7, 2021

DonatoD commented Sep 16, 2021

KirannBhavaraju commented Oct 1, 2021 • edited Loading

Potemkinkuliss commented Oct 27, 2021

SStarosielec commented Nov 28, 2021

Anto79-ops commented Jan 6, 2022 • edited Loading

felice64 commented Jan 8, 2022

Anto79-ops commented Jan 8, 2022 • edited Loading

felice64 commented Jan 9, 2022

Anto79-ops commented Jan 9, 2022

ranimkayali commented May 20, 2022

mmohamed15 commented May 20, 2022

youduda commented Jun 9, 2022

badrbouslikhin commented Jun 10, 2022 • edited Loading

rogerio-pellarin commented Dec 1, 2022

tomy983 commented Dec 11, 2022 • edited Loading

DonatoD commented Dec 12, 2022 • edited Loading

DonatoD commented Dec 12, 2022 • edited Loading

mmohamed15 commented Aug 6, 2021 •

edited

Loading

KirannBhavaraju commented Oct 1, 2021 •

edited

Loading

Anto79-ops commented Jan 6, 2022 •

edited

Loading

Anto79-ops commented Jan 8, 2022 •

edited

Loading

badrbouslikhin commented Jun 10, 2022 •

edited

Loading

tomy983 commented Dec 11, 2022 •

edited

Loading

DonatoD commented Dec 12, 2022 •

edited

Loading

DonatoD commented Dec 12, 2022 •

edited

Loading