

Monitoring

Synse Server does not provide any kind of monitoring solution for the data that it exposes. It does provide basic application-level metrics via a Prometheus exporter.

Configuring¶

By default, application metrics export is disabled. To enable it, you would need to set the following values in the application configuration:

metrics:
  enabled: true

This can also be done via environment variable

SYNSE_METRICS_ENABLED=true

Getting metrics¶

Once configured, you can run Synse Server and verify that metrics are enabled by hitting http://${server}/metrics. If enabled, the route should resolve and you should get Prometheus export data, e.g.

# HELP python_gc_collected_objects Objects collected during gc
# TYPE python_gc_collected_objects histogram
python_gc_collected_objects_bucket{generation="0",le="500.0"} 132.0
python_gc_collected_objects_bucket{generation="0",le="1000.0"} 132.0
python_gc_collected_objects_bucket{generation="0",le="5000.0"} 132.0
python_gc_collected_objects_bucket{generation="0",le="10000.0"} 132.0
python_gc_collected_objects_bucket{generation="0",le="50000.0"} 132.0
python_gc_collected_objects_bucket{generation="0",le="+Inf"} 132.0
python_gc_collected_objects_count{generation="0"} 132.0
python_gc_collected_objects_sum{generation="0"} 1045.0
python_gc_collected_objects_bucket{generation="1",le="500.0"} 12.0
...

Exposed metrics¶

In addition to the default Python metrics (e.g. Python garbage collection, Python info, process memory, etc.), Synse Server exposes a number of metrics which describe its application performance. Below are a summary of those metrics.

HTTP Request Count¶


name	`synse_http_request_count`
description	The total number of HTTP requests processed.
metric type	Counter
labels	`method`, `endpoint`, `http_code`

HTTP Request Latency (seconds)¶


name	`synse_http_request_latency_sec`
description	The time it takes for an HTTP request to be fulfilled.
metric type	Histogram
labels	`method`, `endpoint`, `http_code`

HTTP Response Bytes¶


name	`synse_http_response_bytes`
description	The total number of bytes returned in HTTP API responses.
metric type	Counter
labels	`method`, `endpoint`, `http_code`

WebSocket Request Count¶


name	`synse_websocket_request_count`
description	The total number of WebSocket requests processed.
metric type	Counter
labels	`event`

WebSocket Request Latency (seconds)¶


name	`synse_websocket_request_latency_sec`
description	The time it takes for a WebSocket request to be fulfilled.
metric type	Histogram
labels	`event`

WebSocket Request Bytes¶


name	`synse_websocket_request_bytes`
description	The total number of bytes received from WebSocket requests.
metric type	Counter
labels	`event`

WebSocket Response Bytes¶


name	`synse_websocket_response_bytes`
description	The total number of bytes returned from the WebSocket API.
metric type	Counter
labels	`event`

WebSocket Error Response Count¶


name	`synse_websocket_error_response_count`
description	The total number of error responses returned by the WebSocket API.
metric type	Counter
labels	`event`

WebSocket Session Count¶


name	`synse_websocket_session_count`
description	The total number of active WebSocket sessions connected to Synse Server.
metric type	Gauge
labels	`source`

gRPC Message Sent¶


name	`synse_grpc_message_sent`
description	The total number of gRPC messages sent to plugins.
metric type	Counter
labels	`type`, `service`, `method`, `plugin`

gRPC Message Received¶


name	`synse_grpc_message_received`
description	The total number of gRPC messages received from plugins.
metric type	Counter
labels	`type`, `service`, `method`, `plugin`

gRPC Request Latency (seconds)¶


name	`synse_grpc_request_latency_sec`
description	The time it takes for a gRPC request to be fulfilled.
metric type	Histogram
labels	`type`, `service`, `method`, `plugin`

Viewing application metrics¶

An example deployment is provided in the project's GitHub repository which starts up a Synse Server instance, a Prometheus instance to collect application metrics, and a Grafana instance (with pre-built example dashboard) to visualize those metrics.

From the project source, you can simply run:

# vapor-ware/synse-server/examples/monitoring/
docker-compose up -d

All components will start up and be exposed locally

Service	Address
synse-server	`localhost:5000`
prometheus	`localhost:9090`
grafana	`localhost:3000`

The login for the Grafana dashboard at localhost:3000 is admin/admin. Once logged in, you can select the "Synse Server Application Metrics" dashboard.

To get requests data, you will need to hit Synse Server endpoints, e.g.

localhost:5000/test
localhost:5000/version
localhost:5000/v3/config
localhost:5000/v3/plugin
localhost:5000/v3/scan
...