Skip to content

Commit

Permalink
Connect probers and notifiers by Channel (megaease#108)
Browse files Browse the repository at this point in the history
* [WIP] prober -> channel -> notify

* typo fix

* refactory the code

* write the README.md

* Update README.md

Co-authored-by: Pantelis Roditis <[email protected]>

* fix the concurrent bug

* change the time.After() to time.NewTimer(), and remove the blank line

* remove the blank line

* remove the interface reference

* rename the prober.go to probe.go

* Update README.md

Co-authored-by: Pantelis Roditis <[email protected]>

* Update README.md

Co-authored-by: Pantelis Roditis <[email protected]>

* Update README.md

Co-authored-by: Pantelis Roditis <[email protected]>

Co-authored-by: Pantelis Roditis <[email protected]>
  • Loading branch information
haoel and proditis committed May 27, 2022
1 parent bd8161c commit e6ab189
Show file tree
Hide file tree
Showing 16 changed files with 766 additions and 211 deletions.
2 changes: 1 addition & 1 deletion .goreleaser.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ changelog:

builds:
- id: build
main: cmd/easeprobe/main.go
main: ./cmd/easeprobe/
binary: bin/easeprobe
env:
- CGO_ENABLED=0
Expand Down
70 changes: 60 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ EaseProbe is a simple, standalone, and lightWeight tool that can do health/statu
- [1.1 Probe](#11-probe)
- [1.2 Notification](#12-notification)
- [1.3 Report](#13-report)
- [1.4 Administration](#14-administration)
- [1.5 Prometheus Metrics](#15-prometheus-metrics)
- [1.4 Channel](#14-channel)
- [1.5 Administration](#15-administration)
- [1.6 Prometheus Metrics](#16-prometheus-metrics)
- [2. Getting Started](#2-getting-started)
- [2.1 Build](#21-build)
- [2.2 Configure](#22-configure)
Expand Down Expand Up @@ -143,9 +144,9 @@ Ease Probe supports the following notifications:
- **SMS**. Support SMS notification with multiple SMS service providers - [Twilio](https://www.twilio.com/sms), [Vonage(Nexmo)](https://developer.vonage.com/messaging/sms/overview), [YunPain](https://www.yunpian.com/doc/en/domestic/list.html)
- **Teams**. Support the [Microsoft Teams](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/connectors-using?tabs=cURL#setting-up-a-custom-incoming-webhook) notification.

**Note**:

- The notification is **Edge-Triggered Mode**, only notified while the status is changed.
> **Note**:
>
> The notification is **Edge-Triggered Mode**, only notified while the status is changed.
```YAML
# Notification Configuration
Expand Down Expand Up @@ -237,7 +238,56 @@ Check the [Notification Configuration](#37-notification-configuration) to see h
For more information, please check the [Global Setting Configuration](#38-global-setting-configuration)


### 1.4 Administration
### 1.4 Channel

The Channel is used for connecting the Probers and the Notifiers. It can be configured for every Prober and Notifier.

This feature could help you group the Probers and Notifiers into a logical group.

> **Note**:
>
> 1) If you don't define the Channel, the default channel will be used for these probers and notifiers. The default channel name is `__EaseProbe_Channel__`
>
> 2) Versions of EaseProbe prior to v1.5.0, do not support the `channel` feature
```YAML

For example:

```YAML
http:
- name: probe A
channels : [ Dev_Channel, Manager_Channel ]
shell:
- name: probe B
channels: [ Ops_Channel ]
notify:
- discord: Discord
channels: [ Dev_Channel, Ops_Channel ]
- email: Gmail
channels: [ Mgmt_Channel ]
```

Then, we will have the following diagram

```
┌───────┐ ┌──────────────┐
│Probe B├─────────►│ Mgmt_Channel ├────┐
└───────┘ └──────────────┘ │
┌─────────────┐ │ ┌─────────┐
┌─────►│ Dev_Channel ├─────▼───► Discord │
│ └─────────────┘ └─────────┘
┌───────┐ │
│Probe A├───┤
└───────┘ │
│ ┌────────────┐ ┌─────────┐
└─────►│ QA_Channel ├──────────► Gmail │
└────────────┘ └─────────┘
```
### 1.5 Administration
There are some administration configuration options:
Expand Down Expand Up @@ -290,7 +340,7 @@ There are some administration configuration options:

EaseProbe accepts the `HUP` signal to rotate the log.

### 1.5 Prometheus Metrics
### 1.6 Prometheus Metrics

EaseProbe supports Prometheus metrics. The Prometheus endpoint is `http:https://localhost:8181/metrics` by default.

Expand Down Expand Up @@ -355,7 +405,7 @@ version: v1.5.0

The following example configurations illustrate the EaseProbe supported features.

**Notes**: All probes have the following options:
**Note**: All probes have the following options:

- `timeout` - the maximum time to wait for the probe to complete. default: `30s`.
- `interval` - the interval time to run the probe. default: `1m`.
Expand Down Expand Up @@ -530,7 +580,7 @@ Support the host probe, the configuration example as below.

The feature probe the CPU, Memory, and Disk usage, if one of them exceeds the threshold, then mark the host as status down.

> Note:
> **Note**:
> - The thresholds are **OR** conditions, if one of them exceeds the threshold, then mark the host as status down.
> - The Host needs remote server have the following command: `top`, `df`, `free`, `awk`, `grep`, `tr`, and `hostname` (check the [source code](./probe/host/host.go) to see how it works).
> - The disk usage only check the root disk.
Expand Down Expand Up @@ -684,7 +734,7 @@ notify:

```

**Notes**: All of the notifications can have the following optional configuration.
**Note**: All of the notifications can have the following optional configuration.

```YAML
dry: true # dry notification, print the Discord JSON in log(STDOUT)
Expand Down
164 changes: 164 additions & 0 deletions channel/channel.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
/*
* Copyright (c) 2022, MegaEase
* All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http:https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package channel

import (
"github.com/megaease/easeprobe/notify"
"github.com/megaease/easeprobe/probe"
log "github.com/sirupsen/logrus"
"sync/atomic"
)

const kind = "channel"

// Channel implements a config for Channel
type Channel struct {
Name string `yaml:"name"` // unique name
Probers map[string]probe.Prober `yaml:"probers"` // probers
Notifiers map[string]notify.Notify `yaml:"notifiers"` // notifiers
isWatch int32 `yaml:"-"` // is watch
done chan bool `yaml:"-"` // done channel
channel chan probe.Result `yaml:"-"` // notify channel

}

// NewEmpty creates a new empty Channel object with nil channel
// After setup the probers, You have to call Config() to create the channel
func NewEmpty(name string) *Channel {
return &Channel{
Name: name,
Probers: map[string]probe.Prober{},
Notifiers: map[string]notify.Notify{},
isWatch: 0,
done: nil,
channel: nil,
}
}

// Config configures the channel
func (c *Channel) Config() {
c.done = make(chan bool)
c.channel = make(chan probe.Result, len(c.Probers))
}

// Done returns the done channel
func (c *Channel) Done() chan bool {
return c.done
}

// Channel returns the notification channel
func (c *Channel) Channel() chan probe.Result {
return c.channel
}

// Send sends the result to the channel
func (c *Channel) Send(result probe.Result) {
c.channel <- result
}

// GetProber returns the Notify object
func (c *Channel) GetProber(name string) probe.Prober {
return c.Probers[name]
}

// SetProbers sets the Notify objects
func (c *Channel) SetProbers(probers []probe.Prober) {
for _, p := range probers {
c.SetProber(p)
}
}

// SetProber sets the Notify object
func (c *Channel) SetProber(p probe.Prober) {
if p == nil {
return
}
c.Probers[p.Name()] = p
}

// GetNotify returns the Notify object
func (c *Channel) GetNotify(name string) notify.Notify {
return c.Notifiers[name]
}

// SetNotifiers sets the Notify objects
func (c *Channel) SetNotifiers(notifiers []notify.Notify) {
for _, n := range notifiers {
c.SetNotify(n)
}
}

// SetNotify sets the Notify object
func (c *Channel) SetNotify(n notify.Notify) {
if n == nil {
return
}
c.Notifiers[n.GetName()] = n
}

var dryNotify bool

// SetDryNotify sets the global dry run flag
func SetDryNotify(dry bool) {
dryNotify = dry
}

// WatchEvent watches the notification event
// Go through all of notification to notify the result.
func (c *Channel) WatchEvent() {
// check if the channel is watching
if atomic.CompareAndSwapInt32(&(c.isWatch), 0, 1) == false {
log.Warnf("[%s/ %s]: Channel is already watching!", kind, c.Name)
return
}

// set the channel is not watching
defer func() {
atomic.StoreInt32(&(c.isWatch), 0)
}()

// Watching the Probe Event...
for {
select {
case <-c.done:
log.Infof("[%s / %s]: Received the done signal, channel exiting...", kind, c.Name)
return
case result := <-c.channel:
// if the status has no change, no need notify
if result.PreStatus == result.Status {
log.Debugf("[%s / %s]: %s (%s) - Status no change [%s] == [%s], no notification.",
kind, c.Name, result.Name, result.Endpoint, result.PreStatus, result.Status)
continue
}
if result.PreStatus == probe.StatusInit && result.Status == probe.StatusUp {
log.Debugf("[%s / %s]: %s (%s) - Initial Status [%s] == [%s], no notification.",
kind, c.Name, result.Name, result.Endpoint, result.PreStatus, result.Status)
continue
}
log.Infof("[%s / %s]: %s (%s) - Status changed [%s] ==> [%s]",
kind, c.Name, result.Name, result.Endpoint, result.PreStatus, result.Status)
for _, n := range c.Notifiers {
if dryNotify {
n.DryNotify(result)
} else {
go n.Notify(result)
}
}
}
}
}
Loading

0 comments on commit e6ab189

Please sign in to comment.