Skip to content

Commit

Permalink
introduce tls probe (megaease#110)
Browse files Browse the repository at this point in the history
* introduce tls probe

* load str instead of byte

* add readme

* fix type

* add metrics

* metric inspired from blackbox

* happy lint

* add comments

* update the README.md

Co-authored-by: Hao Chen <[email protected]>
  • Loading branch information
tg123 and haoel committed May 27, 2022
1 parent 2ef8571 commit 03f2835
Show file tree
Hide file tree
Showing 5 changed files with 520 additions and 17 deletions.
66 changes: 49 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,11 @@ EaseProbe is a simple, standalone, and lightWeight tool that can do health/statu
- [3.2 TCP Probe Configuration](#32-tcp-probe-configuration)
- [3.3 Shell Command Probe Configuration](#33-shell-command-probe-configuration)
- [3.4 SSH Command Probe Configuration](#34-ssh-command-probe-configuration)
- [3.5 Host Resource Usage Probe Configuration](#35-host-resource-usage-probe-configuration)
- [3.6 Native Client Probe Configuration](#36-native-client-probe-configuration)
- [3.7 Notification Configuration](#37-notification-configuration)
- [3.8 Global Setting Configuration](#38-global-setting-configuration)
- [3.5 TLS Probe Configuration](#35-tls-probe-configuration)
- [3.6 Host Resource Usage Probe Configuration](#36-host-resource-usage-probe-configuration)
- [3.7 Native Client Probe Configuration](#37-native-client-probe-configuration)
- [3.8 Notification Configuration](#38-notification-configuration)
- [3.9 Global Setting Configuration](#39-global-setting-configuration)
- [4. Community](#4-community)
- [5. License](#5-license)

Expand Down Expand Up @@ -94,7 +95,15 @@ Ease Probe supports the following probing methods: **HTTP**, **TCP**, **Shell Co
contain: easeprobe
```

- **Host**. Run an SSH command on a remote host and check the CPU, Memory, and Disk usage. ( [Host Load Probe](#35-host-resource-usage-probe-configuration) )
- **TLS**. Ping the remote endpoint, can probe for revoked or expired certificates ( [TLS Probe Configuration](#35-tls-probe-configuration) )

```YAML
tls:
- name: expired test
host: expired.badssl.com:443
```

- **Host**. Run an SSH command on a remote host and check the CPU, Memory, and Disk usage. ( [Host Load Probe](#36-host-resource-usage-probe-configuration) )

```yaml
host:
Expand All @@ -108,7 +117,7 @@ Ease Probe supports the following probing methods: **HTTP**, **TCP**, **Shell Co
disk: 0.90 # disk usage 90%
```

- **Client**. Currently, support the following native client. Support the mTLS. (refer to: [Native Client Probe Configuration](#36-native-client-probe-configuration) )
- **Client**. Currently, support the following native client. Support the mTLS. (refer to: [Native Client Probe Configuration](#37-native-client-probe-configuration) )
- **MySQL**. Connect to the MySQL server and run the `SHOW STATUS` SQL.
- **Redis**. Connect to the Redis server and run the `PING` command.
- **MongoDB**. Connect to MongoDB server and just ping server.
Expand Down Expand Up @@ -195,7 +204,7 @@ notify:
webhook: "https://outlook.office365.com/webhook/a1269812-6d10-44b1-abc5-b84f93580ba0@9e7b80c7-d1eb-4b52-8582-76f921e416d9/IncomingWebhook/3fdd6767bae44ac58e5995547d66a4e4/f332c8d9-3397-4ac5-957b-b8e3fc465a8c" # see https://docs.microsoft.com/en-us/outlook/actionable-messages/send-via-connectors
```

Check the [Notification Configuration](#37-notification-configuration) to see how to configure it.
Check the [Notification Configuration](#38-notification-configuration) to see how to configure it.

### 1.3 Report

Expand All @@ -218,14 +227,14 @@ Check the [Notification Configuration](#37-notification-configuration) to see h
- HTML: `http:https://localhost:8181/` or `http:https://localhost:8181/?refresh=30s`
- JSON: `http:https://localhost:8181/api/v1/sla`

Refer to the [Global Setting Configuration](#38-global-setting-configuration) to see how to configure the access log.
Refer to the [Global Setting Configuration](#39-global-setting-configuration) to see how to configure the access log.


- **SLA Data Persistence**. Save the SLA statistics data on the disk.

The SLA data would be persisted in `$CWD/data/data.yaml` by default. If you want to configure the path, you can do it in the `settings` section.

When EaseProbe starts, it looks for the location of `data.yaml` and if found, load the file and remove any probes that are no longer present in the configuration file. Setting a value of `"-"` for `data:` disables SLA persistence (eg `data: "-"`).
When EaseProbe starts, it looks for the location of `data.yaml` and if found, loads the file and removes any probes that are no longer present in the configuration file. Setting a value of `"-"` for `data:` disables SLA persistence (eg `data: "-"`).

```YAML
settings:
Expand Down Expand Up @@ -312,7 +321,7 @@ There are some administration configuration options:

**2) Log file Rotation**

There are two types of log file: **Application Log** and **HTTP Access Log**.
There are two types of the log files: **Application Log** and **HTTP Access Log**.

Both Application Log and HTTP Access Log would be StdOut by default. They all can be configured by:

Expand Down Expand Up @@ -353,9 +362,9 @@ Refer to the [Global Setting Configuration](#38-global-setting-configuration) to
## 2. Getting Started

You can get started with EaseProbe, by any of the following methods:
* download the release for your platform from https://github.com/megaease/easeprobe/releases
* use the available EaseProbe docker image `docker run -it megaease/easeprobe`
* build `easeprobe` from sources
* Download the release for your platform from https://github.com/megaease/easeprobe/releases
* Use the available EaseProbe docker image `docker run -it megaease/easeprobe`
* Build `easeprobe` from sources

### 2.1 Build

Expand Down Expand Up @@ -574,7 +583,29 @@ ssh:
cmd: "ps -ef | grep kafka"
```

### 3.5 Host Resource Usage Probe Configuration
### 3.5 TLS Probe Configuration

TLS ping to remote endpoint, can probe for revoked or expired certificates

```YAML
tls:
- name: expired test
host: expired.badssl.com:443
insecure_skip_verify: true # dont check cert validity
expire_skip_verify: true # dont check cert expire date
# root_ca_pem_path: /path/to/root/ca.pem # ignore if root_ca_pem is present
# root_ca_pem: |
# -----BEGIN CERTIFICATE-----
- name: untrust test
host: untrusted-root.badssl.com:443
# insecure_skip_verify: true # dont check cert validity
# expire_skip_verify: true # dont check cert expire date
# root_ca_pem_path: /path/to/root/ca.pem # ignore if root_ca_pem is present
# root_ca_pem: |
# -----BEGIN CERTIFICATE-----
```

### 3.6 Host Resource Usage Probe Configuration

Support the host probe, the configuration example as below.

Expand Down Expand Up @@ -609,7 +640,7 @@ host:
key: /Users/user/.ssh/id_rsa
```

### 3.6 Native Client Probe Configuration
### 3.7 Native Client Probe Configuration

```YAML
# Native Client Probe
Expand Down Expand Up @@ -661,7 +692,7 @@ client:
```


### 3.7 Notification Configuration
### 3.8 Notification Configuration

```YAML
# Notification Configuration
Expand Down Expand Up @@ -745,7 +776,7 @@ notify:
```


### 3.8 Global Setting Configuration
### 3.9 Global Setting Configuration

```YAML
# Global settings for all probes and notifiers.
Expand Down Expand Up @@ -850,3 +881,4 @@ settings:
## 5. License

EaseProbe is under the Apache 2.0 license. See the [LICENSE](./LICENSE) file for details.

3 changes: 3 additions & 0 deletions conf/conf.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import (
"github.com/megaease/easeprobe/probe/shell"
"github.com/megaease/easeprobe/probe/ssh"
"github.com/megaease/easeprobe/probe/tcp"
"github.com/megaease/easeprobe/probe/tls"

log "github.com/sirupsen/logrus"
"gopkg.in/yaml.v3"
Expand Down Expand Up @@ -134,6 +135,7 @@ type Conf struct {
Shell []shell.Shell `yaml:"shell"`
Client []client.Client `yaml:"client"`
SSH ssh.SSH `yaml:"ssh"`
TLS []tls.TLS `yaml:"tls"`
Host host.Host `yaml:"host"`
Notify notify.Config `yaml:"notify"`
Settings Settings `yaml:"settings"`
Expand Down Expand Up @@ -206,6 +208,7 @@ func New(conf *string) (*Conf, error) {
Bastion: &ssh.BastionMap,
Servers: []ssh.Server{},
},
TLS: []tls.TLS{},
Host: host.Host{
Bastion: &host.BastionMap,
Servers: []host.Server{},
Expand Down
71 changes: 71 additions & 0 deletions probe/tls/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
/*
* Copyright (c) 2022, MegaEase
* All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http:https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package tls

import (
"crypto/tls"
"time"

"github.com/megaease/easeprobe/global"
"github.com/megaease/easeprobe/metric"
"github.com/prometheus/client_golang/prometheus"
)

// code and metric idea from https://github.com/prometheus/blackbox_exporter/blob/master/prober/tls.go
type metrics struct {
EarliestCertExpiry *prometheus.GaugeVec
LastChainExpiryTimestampSeconds *prometheus.GaugeVec
}

// newMetrics create the HTTP metrics
func newMetrics(subsystem, name string) *metrics {
namespace := global.GetEaseProbe().Name
return &metrics{
EarliestCertExpiry: metric.NewGauge(namespace, subsystem, name, "earliest_cert_expiry",
"last TLS chain expiry in timestamp seconds", []string{}),
LastChainExpiryTimestampSeconds: metric.NewGauge(namespace, subsystem, name, "last_chain_expiry_timestamp_seconds",
"earliest TLS cert expiry in unixtime", []string{}),
}
}

func getEarliestCertExpiry(state *tls.ConnectionState) time.Time {
earliest := time.Time{}
for _, cert := range state.PeerCertificates {
if (earliest.IsZero() || cert.NotAfter.Before(earliest)) && !cert.NotAfter.IsZero() {
earliest = cert.NotAfter
}
}
return earliest
}

func getLastChainExpiry(state *tls.ConnectionState) time.Time {
lastChainExpiry := time.Time{}
for _, chain := range state.VerifiedChains {
earliestCertExpiry := time.Time{}
for _, cert := range chain {
if (earliestCertExpiry.IsZero() || cert.NotAfter.Before(earliestCertExpiry)) && !cert.NotAfter.IsZero() {
earliestCertExpiry = cert.NotAfter
}
}
if lastChainExpiry.IsZero() || lastChainExpiry.Before(earliestCertExpiry) {
lastChainExpiry = earliestCertExpiry
}

}
return lastChainExpiry
}
131 changes: 131 additions & 0 deletions probe/tls/tls.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
/*
* Copyright (c) 2022, MegaEase
* All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http:https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package tls

import (
"context"
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"net"
"strings"
"time"

"github.com/megaease/easeprobe/global"
"github.com/megaease/easeprobe/probe/base"
"github.com/prometheus/client_golang/prometheus"

log "github.com/sirupsen/logrus"
)

// TLS implements a config for TLS
type TLS struct {
base.DefaultOptions `yaml:",inline"`
Host string `yaml:"host"`
InsecureSkipVerify bool `yaml:"insecure_skip_verify"`

RootCAPemPath string `yaml:"root_ca_pem_path"`
RootCaPem string `yaml:"root_ca_pem"`
rootCAs *x509.CertPool

ExpireSkipVerify bool `yaml:"expire_skip_verify"`

metrics *metrics
}

// Config HTTP Config Object
func (t *TLS) Config(gConf global.ProbeSettings) error {
kind := "tls"
tag := ""
name := t.ProbeName
t.DefaultOptions.Config(gConf, kind, tag, name, t.Host, t.DoProbe)

rootCaPem := []byte(t.RootCaPem)

if len(rootCaPem) == 0 && t.RootCAPemPath != "" {
var err error
rootCaPem, err = ioutil.ReadFile(t.RootCAPemPath)
if err != nil {
return err
}
}

if len(rootCaPem) > 0 {
t.rootCAs = x509.NewCertPool()
if !(t.rootCAs.AppendCertsFromPEM(rootCaPem)) {
return fmt.Errorf("cannot parse root ca pem")
}
}

t.metrics = newMetrics(kind, tag)

log.Debugf("[%s] configuration: %+v, %+v", t.ProbeKind, t, t.Result())
return nil
}

// DoProbe return the checking result
func (t *TLS) DoProbe() (bool, string) {
addr := t.Host
conn, err := net.DialTimeout("tcp", addr, t.Timeout())
if err != nil {
log.Errorf("tcp dial error: %v", err)
return false, fmt.Sprintf("tcp dial error: %v", err)
}
defer conn.Close()

colonPos := strings.LastIndex(addr, ":")
if colonPos == -1 {
colonPos = len(addr)
}
hostname := addr[:colonPos]

tconn := tls.Client(conn, &tls.Config{
InsecureSkipVerify: t.InsecureSkipVerify,
RootCAs: t.rootCAs,
ServerName: hostname,
})

ctx, cancel := context.WithTimeout(context.Background(), t.Timeout())
defer cancel()
err = tconn.HandshakeContext(ctx)
if err != nil {
log.Errorf("tls handshake error: %v", err)
return false, fmt.Sprintf("tls handshake error: %v", err)
}

if !t.ExpireSkipVerify {
for _, cert := range tconn.ConnectionState().PeerCertificates {
valid := true
valid = valid && !time.Now().After(cert.NotAfter)
valid = valid && !time.Now().Before(cert.NotBefore)

if !valid {
log.Errorf("host %v cert expired", t.Host)
return false, "certificate is expired or not yet valid"
}
}
}

state := tconn.ConnectionState()

t.metrics.EarliestCertExpiry.With(prometheus.Labels{}).Set(float64(getEarliestCertExpiry(&state).Unix()))
t.metrics.LastChainExpiryTimestampSeconds.With(prometheus.Labels{}).Set(float64(getLastChainExpiry(&state).Unix()))

return true, "TLS Endpoint Verified Successfully!"
}
Loading

0 comments on commit 03f2835

Please sign in to comment.