csdsclient: add visualization #15

I really like this feature. I'm a little unclear on the details though. What questions is the visualization intended to solve? Can you describe the rules used to build the graphviz markup? Does color represent sync status? perhaps a key section would be useful...

Rainton · 2020-08-04T00:36:41Z

@therealmitchconnors Sorry for my unclear expression.

The detailed config in response is always so long that users cannot figure out the relationships between each xDS easily. The visualization feature is used to give users a clear view of the relationships by going through the detailed config, storing relationships between xDS in maps, drawing a graph by marking xDS as nodes and relationships as edges. Different colors represent different xDS types, such as LDS, RDS, CDS, and EDS.

Maybe you can take a look at this sample graph to get clearer to what this feature would be like.

The user scenario may be like this: the user sees the graph and finds that there are two RDSs here and both of them are related to the CDS0. If the user want to see the detailed config of one of the RDS, he can just hovers the mouse over the specific RDS node and its real name will shows. Then the user can find the config of this RDS easily by searching this real name in the detailed config.

Signed-off-by: Yutong Li <[email protected]>

fuqianggao · 2020-08-05T23:23:37Z

csds-client/README.md

@@ -10,7 +10,7 @@ For now, this initial version of this CSDS client only supports GCP's [Traffic D
 # Running
 * run with `./bin/csds <flag>`, e.g. <br/><br/>
 * auto authentication mode
- ```
+ ```bash
 ./bin/csds \
 -service_uri <uri> \


maybe consider change this to csds-client?

I've modified the Makefile to run go install to build the binary under GOPATH/bin. In this way, the binary will be renamed as csds-client and we can just run it with csds-client <flag>

csds-client/README.md

csds-client/client/client.go

fuqianggao · 2020-08-05T23:45:31Z

csds-client/client/util.go

+ rds := make(map[string]string)
+ cds := make(map[string]string)
+ ldsToRds := make(map[string]*treeset.Set)
+ rdsToCds := make(map[string]*treeset.Set)


I think we should also handle cds to eds

What about adding handler for cds to eds in the v3 PR? Since for v2, there should not be eds in the detailed config?

Signed-off-by: Yutong Li <[email protected]>

dio

Few comments. And it can be cleaned up in the next iteration.

csds-client/client/util.go

dio · 2020-08-06T00:42:05Z

csds-client/client/util.go

+ "bytes"
+ "encoding/json"
+ "fmt"
+ "io"
+ "io/ioutil"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "runtime"
+ "strconv"
+
+ "github.com/awalterschulze/gographviz"
+ "github.com/emirpasic/gods/sets/treeset"
 envoy_api_v2 "github.com/envoyproxy/go-control-plane/envoy/api/v2"
 envoy_config_filter_http_router_v2 "github.com/envoyproxy/go-control-plane/envoy/config/filter/http/router/v2"
 envoy_config_filter_network_http_connection_manager_v2 "github.com/envoyproxy/go-control-plane/envoy/config/filter/network/http_connection_manager/v2"
+ csdspb "github.com/envoyproxy/go-control-plane/envoy/service/status/v2"
 envoy_type_matcher "github.com/envoyproxy/go-control-plane/envoy/type/matcher"


I'll enable a linter soon. So hopefully sorting this import will be "standardized".

Cool! Thanks a lot!

csds-client/client/response_for_visualization.json

dio · 2020-08-06T00:43:06Z

csds-client/client/client_test.go

+ if err != nil {
+ t.Errorf("Generate graph failure:%v", err)
+ }
+ want := "digraph G {\nrankdir=LR;\n\\\"test_lds_0\\\"->\\\"test_rds_0\\\"[ arrowsize=0.3, penwidth=0.3 ];\n\\\"test_lds_0\\\"->\\\"test_rds_1\\\"[ arrowsize=0.3, penwidth=0.3 ];\n\\\"test_rds_0\\\"->\\\"test_cds_0\\\"[ arrowsize=0.3, penwidth=0.3 ];\n\\\"test_rds_0\\\"->\\\"test_cds_1\\\"[ arrowsize=0.3, penwidth=0.3 ];\n\\\"test_rds_1\\\"->\\\"test_cds_1\\\"[ arrowsize=0.3, penwidth=0.3 ];\n\\\"test_cds_0\\\" [ color=\\\"#34A853\\\", fillcolor=\\\"#34A853\\\", fontcolor=white, fontname=Roboto, label=CDS0, shape=box, style=\\\"\"filled,rounded\"\\\" ];\n\\\"test_cds_1\\\" [ color=\\\"#34A853\\\", fillcolor=\\\"#34A853\\\", fontcolor=white, fontname=Roboto, label=CDS1, shape=box, style=\\\"\"filled,rounded\"\\\" ];\n\\\"test_lds_0\\\" [ color=\\\"#4285F4\\\", fillcolor=\\\"#4285F4\\\", fontcolor=white, fontname=Roboto, label=LDS0, shape=box, style=\\\"\"filled,rounded\"\\\" ];\n\\\"test_rds_0\\\" [ color=\\\"#FBBC04\\\", fillcolor=\\\"#FBBC04\\\", fontcolor=white, fontname=Roboto, label=RDS0, shape=box, style=\\\"\"filled,rounded\"\\\" ];\n\\\"test_rds_1\\\" [ color=\\\"#FBBC04\\\", fillcolor=\\\"#FBBC04\\\", fontcolor=white, fontname=Roboto, label=RDS1, shape=box, style=\\\"\"filled,rounded\"\\\" ];\n\n}\n"


Have a comment to "manually" review this? A link to that viewer site with this data?

I modified this test to open two windows in the browser to show the expected graph and the generated graph. Does this make sense?

csds-client/client/client_test.go

csds-client/client/client.go

Signed-off-by: Yutong Li <[email protected]>

printoutResponse() to info Signed-off-by: Yutong Li <[email protected]>

dio · 2020-08-07T02:10:14Z

Seems like you need to merge master.

Rainton · 2020-08-07T03:48:30Z

@dio Sorry about that. I've just merged.

Rainton added 30 commits July 9, 2020 00:50

add a simple client

cec3c6a

Signed-off-by: Yutong Li <[email protected]>

add sending request

f8121a9

Signed-off-by: Yutong Li <[email protected]>

add NodeMetadatas in NodeMatcher

f539265

Signed-off-by: Yutong Li <[email protected]>

modify project number and network name to send a correct

559fd96

ClientStatusRequest Signed-off-by: Yutong Li <[email protected]>

parse flags

1a40c70

Signed-off-by: Yutong Li <[email protected]>

add yaml parser

a93209e

Signed-off-by: Yutong Li <[email protected]>

replace yaml parser to protojson.unmarshal

b16dbf7

Signed-off-by: Yutong Li <[email protected]>

add auto generated creds by ADC

c049867

Signed-off-by: Yutong Li <[email protected]>

add error handler to unsupported auto authn_mode platforms

cb3a2a9

Signed-off-by: Yutong Li <[email protected]>

add proprocessing response

bbe727d

Signed-off-by: Yutong Li <[email protected]>

add formatted output

cf1a626

Signed-off-by: Yutong Li <[email protected]>

add go mod

f91c902

Signed-off-by: Yutong Li <[email protected]>

add test

19d46b0

Signed-off-by: Yutong Li <[email protected]>

add flag check

e289f96

Signed-off-by: Yutong Li <[email protected]>

add README

fa412ee

Signed-off-by: Yutong Li <[email protected]>

update README

e6370e9

Signed-off-by: Yutong Li <[email protected]>

add comments

51c3edc

Signed-off-by: Yutong Li <[email protected]>

add a simple client

254f139

Signed-off-by: Yutong Li <[email protected]>

add flag request_yaml

9f17bf4

Signed-off-by: Yutong Li <[email protected]>

modify default output to stdout

ff7e5bc

Signed-off-by: Yutong Li <[email protected]>

update README

7436b78

Signed-off-by: Yutong Li <[email protected]>

add -monitor_freq flag

e21f641

solved some format issue Signed-off-by: Yutong Li <[email protected]>

update README

7f73cec

Signed-off-by: Yutong Li <[email protected]>

update README

d12c924

Signed-off-by: Yutong Li <[email protected]>

modified return errors

230b7df

Signed-off-by: Yutong Li <[email protected]>

add isJson() to validate json string

bc8d96e

Signed-off-by: Yutong Li <[email protected]>

changed parseGCPProjectNumber() to getValueByKeyFromNodeMatcher() to

14ba384

make it reusable Signed-off-by: Yutong Li <[email protected]>

add check for required fields in request yaml

09ebcdb

Signed-off-by: Yutong Li <[email protected]>

fix format issue

138442c

Signed-off-by: Yutong Li <[email protected]>

update import

9a0cec2

Signed-off-by: Yutong Li <[email protected]>

Rainton added 12 commits July 20, 2020 23:24

Merge branch 'feature-branch' into graph-branch

83d38cf

Signed-off-by: Yutong Li <[email protected]>

add flag -visualization and store the .dot file locally

bbcf189

Signed-off-by: Yutong Li <[email protected]>

add test TestVisualization, fix space problem in test

33dc08d

Signed-off-by: Yutong Li <[email protected]>

beautify the graph

37171cd

Signed-off-by: Yutong Li <[email protected]>

disable opening browser while running with monitor and visualization

5e766d8

mode together Signed-off-by: Yutong Li <[email protected]>

add Makefile and update README

354e548

Signed-off-by: Yutong Li <[email protected]>

Merge branch 'feature-branch' into graph-branch

6ffe46a

Signed-off-by: Yutong Li <[email protected]>

solve conflict

9eff465

Signed-off-by: Yutong Li <[email protected]>

Merge branch 'master' into graph-branch

cfdeef5

Signed-off-by: Yutong Li <[email protected]>

fix conflict

d783ecc

Signed-off-by: Yutong Li <[email protected]>

update bash in README

ba8c933

Signed-off-by: Yutong Li <[email protected]>

update test

82ead6a

Signed-off-by: Yutong Li <[email protected]>

Rainton added 3 commits August 5, 2020 04:13

Merge branch 'master' into graph-branch

b837aae

Signed-off-by: Yutong Li <[email protected]>

fix format issue

56522ab

Signed-off-by: Yutong Li <[email protected]>

fix grammar issue

a8d95d9

Signed-off-by: Yutong Li <[email protected]>

fuqianggao reviewed Aug 5, 2020

View reviewed changes

update comments and README

d56d194

Signed-off-by: Yutong Li <[email protected]>

dio reviewed Aug 6, 2020

View reviewed changes

Rainton added 2 commits August 6, 2020 03:37

update Makefile to build the binary under GOPATH/bin

cc3257f

Signed-off-by: Yutong Li <[email protected]>

replace the visualization test to manual check, replace the params of

1b65373

printoutResponse() to info Signed-off-by: Yutong Li <[email protected]>

dio approved these changes Aug 7, 2020

View reviewed changes

Merge branch 'master' into graph-branch

b9a8b66

dio merged commit f6b73a4 into envoyproxy:master Aug 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

csdsclient: add visualization #15

csdsclient: add visualization #15

Rainton commented Jul 31, 2020 •

edited

Loading

Rainton commented Jul 31, 2020

fuqianggao commented Jul 31, 2020 •

edited

Loading

therealmitchconnors commented Aug 3, 2020

Rainton commented Aug 4, 2020

fuqianggao Aug 5, 2020

dio Aug 6, 2020

Rainton Aug 6, 2020

fuqianggao Aug 5, 2020

Rainton Aug 6, 2020

fuqianggao Aug 6, 2020

dio left a comment

dio Aug 6, 2020

Rainton Aug 6, 2020

dio Aug 6, 2020

Rainton Aug 6, 2020

dio commented Aug 7, 2020

Rainton commented Aug 7, 2020

csdsclient: add visualization #15

csdsclient: add visualization #15

Conversation

Rainton commented Jul 31, 2020 • edited Loading

Rainton commented Jul 31, 2020

fuqianggao commented Jul 31, 2020 • edited Loading

therealmitchconnors commented Aug 3, 2020

Rainton commented Aug 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio commented Aug 7, 2020

Rainton commented Aug 7, 2020

Rainton commented Jul 31, 2020 •

edited

Loading

fuqianggao commented Jul 31, 2020 •

edited

Loading