🛰️ A sophisticated network mapper and analyser
IMPORTANT NOTICE: This repository serves as a demonstration of the confidential project "Wiremap" and is meant for showcase purposes only. We have taken precautions to remove any sensitive information such as academic papers, screenshots, source code, and specific technical details from this repository.
- 1. Abstract
- 2. Showcase video
- 3. System architecture
- 4. Tech stack
- 5. Getting started
- 6. Wiremap inner workings
- 7. Additional material
- 8. Directory structure
- 9. Additional tools
Wiremap is a comprehensive, self-contained network traffic analysis system developed to meet the challenges posed by the increasing complexity of networks. The main objective of the project was to provide a high-quality tool capable of analyzing large amounts of network data, providing an overview of the network topology and its components.
The system was designed to ensure a highly scalable and maintainable architecture capable of delivering high performance. To achieve this goal, we chose to use technologies such as Go, gRPC, Keycloak, and GraphQL, which allowed us to implement a microservice architecture that can guarantee high availability and sustain any traffic peaks.
The results obtained show that Wiremap can analyze large amounts of data accurately and quickly, providing a detailed map of the network. In addition, due to the microservice architecture design chosen, the system is highly scalable and can be easily expanded with new features.
In conclusion, Wiremap is a high-quality and secure tool that allows both experienced and novice users to get an overview of a network, identifying its different hosts and the packets they send. The logical relationships among the various hosts in the network are represented by an interactive graph, which gives the end user a powerful tool to visualize the information in a direct and meaningful way.
Keywords: wiremap, network analysis, microservices, scalability, security
IMPORTANT NOTICE: Due to the confidential nature of the project, we redacted the video to remove any sensitive information. Please, refer to the showcase directory to have a glimpse of the project.
The project is divided into two parts: the frontend and the backend. The frontend is a Next.js application written in TypeScript and the backend is a set of microservices written in Go. The communication between the frontend and the backend is done through a GraphQL Federation gateway, which provides a single API endpoint for all clients. This approach simplifies the communication between clients and services and makes it easier to scale the system horizontally as needed. The authentication is done through Keycloak, an open-source identity and access management solution.
In the frontend, we use React with TypeScript and Chakra UI to build a modern and responsive user interface. The communication with the backend is done through Apollo Client, which provides a simple and powerful way to query the data. To cache the data, we use Stale While Revalidate, which provides a great user experience by showing the cached data while fetching the latest data from the server.
The backend has a microservice architecture, with two services:
Our project is a set of services that are written entirely in Go. The communication between clients and services is done through a GraphQL Federation gateway, which provides a single API endpoint for all clients. This approach simplifies the communication between clients and services and makes it easier to scale the system horizontally as needed.
To create the federation, we use Apollo's Rover CLI to build a supergraph. This supergraph is self-hosted using Apollo Router, which allows us to easily scale the services horizontally by adding more replicas, as well as add new services without having to change the clients. It also masks the internal architecture of the services from the clients, providing an extra layer of security.
To ensure secure authentication and authorization of our users, we use Keycloak - an open source Identity and Access Management solution. The authentication is done using OpenID Connect, which provides a secure and easy-to-use authentication mechanism. The authorization is done using Keycloak's Authorization Services, which provide fine-grained access control to our services.
By using these technologies, we have created a scalable, secure and maintainable architecture for our project. We hope that this documentation will help you understand our project better.
In addition, all the GraphQL queries follows relay.dev Cursor Connections Specification to provide a standardized way to paginate through the data.
Decodes PCAP files and stores the extracted packets into ElasticSearch. It also provides a GraphQL API to query the data.
Tech stack:
Keeps track of the scanned PCAP files and the hosts that were found during the scan. It also provides a GraphQL API to query the data.
Tech stack:
The API gateway is a GraphQL Federation gateway that provides a single API endpoint for all the clients. The router is self-hosted using Apollo Router which provides a precompiled Rust binary that runs as a standalone server. It includes a built-in OpenTelemetry collector that can be used to collect metrics and traces from the services and also Apollo Studio Explorer which provides a web interface to explore and query the supergraph.
While traditional JWT tokens can provide a simple and lightweight way to handle user authentication and authorization, they can be difficult to manage and secure at scale. Keycloak provides a centralized and scalable solution that simplifies user management and enables fine-grained access control across our services (see Keycloak documentation).
With Keycloak, we can easily set up and manage user accounts, groups, and roles, and define policies for user authentication and authorization. Keycloak also supports various authentication mechanisms, including username/password, social logins, and multi-factor authentication, making it easier to implement secure and user-friendly authentication in our services.
Another advantage of Keycloak is that it supports OAuth 2.0 and OpenID Connect protocols, which are widely used in modern web applications and provide additional security features like token revocation and refresh tokens. Keycloak also provides integration with external identity providers like Google, Facebook, and GitHub (see Server Administration Guide), making it easier to implement single sign-on (SSO) across our services.
In summary, Keycloak provides a comprehensive and secure solution for user authentication and authorization, with features like centralized user management, fine-grained access control, and support for modern authentication protocols. These features make it easier to implement secure and scalable authentication in our services, which is why we chose to use Keycloak in our project.
Please, follow the Getting Started Guide to run the project.
The guide will walk you through the process of setting up the project and running it locally.
In this section, we will describe the inner workings of Wiremap. We will start by describing how we integrated Keycloak with the client and microservices. Then, we will describe how we implemented the file upload process and, at last, will be described how a file scan is performed in Wiremap.
This diagram shows the resulting GraphQL schema after the federation process:
As we can see, the decoder and recorder services are federated to the router service. The router service is the only service exposed to the client. The router service is responsible for routing the requests to the correct service.
IMPORTANT NOTICE: Some information has been redacted from the diagram to protect the confidentiality of the project.
This diagram shows the GraphQL schema of the decoder service:
IMPORTANT NOTICE: Some information has been redacted from the diagram to protect the confidentiality of the project.
This diagram shows the GraphQL schema of the recorder service:
IMPORTANT NOTICE: Some information has been redacted from the diagram to protect the confidentiality of the project.
In Wiremap, the clients authenticates using OpenID Connect protocol against wiremap realm in Keycloak.
In the following diagram, we can see the authentication flow of a client interacting with a microservice through the API Gateway:
To optimize file uploading in the decoder service, we use a two-step process inspired by Google and Amazon APIs. When a user wants to upload a file, they first send a request to the API Gateway with the file metadata. The API Gateway then returns a signed URL, which the user can use to upload the file directly to the decoder service. This approach has two main advantages:
-
The API Gateway is not required to handle large files, as the user uploads the file directly to a decoder microservice instance
-
There is no need to store the file in the API Gateway or any other storage service since the file is uploaded directly to the decoder service where it will later be processed.
The following diagram shows the two-step process for uploading a file:
Apollo published a blog post about the best practices for file uploading in GraphQL-based services.
The following diagram provides an overview of the decoding, scanning, and indexing process of a PCAP file. Once the client uploads the file, it sends a startScan mutation to the decoder service, specifying the file ID that the user wants to scan. The router then dispatches this request to the decoder service, which begins the decoding process.
To ensure secure access, the decoder service validates the user session token, using the Keycloak JWKs endpoint as part of its OpenID Connect configuration endpoint. Once validated, the decoder service retrieves the file path from MongoDB and initiates the decoding process using the gopacket library. During this process, the decoder service extracts all unique hosts and their packets, and records the scan by sending a gRPC RecordScan message to the recorder service. The recorder service then validates the user session token, stores the scan in PostgreSQL, and returns the scan ID to the decoder service.
Next, the decoder service records each host by sending a gRPC RecordHost message to the recorder service. After validation, the recorder service stores the host in PostgreSQL and returns the host ID to the decoder service. Each packet is then linked to its corresponding host ID (returned by the recorder microservice) and saved in ElasticSearch using the Bulk API.
Once the decoding process is complete, a network graph is generated and saved in MongoDB using the specified format:
- User guides
- Academic material
- Paper: Redacted
- Poster: Redacted
- Presentation: Redacted
The following diagram shows the directory structure of the project:
.
├── client # Client web application (Next.js + Typescript + ChakraUI)
├── decoder # Decoder microservice (Golang + gRPC + GraphQL)
├── recorder # Recorder microservice (Golang + gRPC + gRPC)
├── router # API Gateway (Apollo Federation 2.0 + Apollo Router)
├── keycloak # Keycloak (Identity and Access Management)
├── grpc # gRPC proto files and Go code generation
└── doc # Documentation (diagrams, etc.)- Snyk - Snyk is a developer-first platform that helps developers use open source code and stay secure. Snyk’s CLI and CI/CD plugins identify vulnerabilities and license issues in open source dependencies, helping developers fix them early in the development process.