mirror of
https://github.com/awatertrevi/infisical.git
synced 2026-10-06 08:27:22 +00:00
misc: improved monitoring and telemetry docs
This commit is contained in:
Sheen Capadngan
@@ -27,7 +27,9 @@ Both approaches provide the same metrics data in OTEL format, so you can choose
|
|||||||
- Access to deploy monitoring services (Prometheus, Grafana, etc.)
|
- Access to deploy monitoring services (Prometheus, Grafana, etc.)
|
||||||
- Basic understanding of Prometheus and Grafana
|
- Basic understanding of Prometheus and Grafana
|
||||||
|
|
||||||
## Environment Variables
|
## Setup
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
Configure the following environment variables in your Infisical backend:
|
Configure the following environment variables in your Infisical backend:
|
||||||
|
|
||||||
@@ -37,287 +39,282 @@ OTEL_TELEMETRY_COLLECTION_ENABLED=true
|
|||||||
|
|
||||||
# Choose export type: "prometheus" or "otlp"
|
# Choose export type: "prometheus" or "otlp"
|
||||||
OTEL_EXPORT_TYPE=prometheus
|
OTEL_EXPORT_TYPE=prometheus
|
||||||
|
|
||||||
# For OTLP push mode, also configure:
|
|
||||||
# OTEL_EXPORT_OTLP_ENDPOINT=http://otel-collector:4318/v1/metrics
|
|
||||||
# OTEL_COLLECTOR_BASIC_AUTH_USERNAME=your_collector_username
|
|
||||||
# OTEL_COLLECTOR_BASIC_AUTH_PASSWORD=your_collector_password
|
|
||||||
# OTEL_OTLP_PUSH_INTERVAL=30000
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Note**: The `OTEL_COLLECTOR_BASIC_AUTH_USERNAME` and `OTEL_COLLECTOR_BASIC_AUTH_PASSWORD` values must match the credentials configured in your OpenTelemetry Collector's `basicauth/server` extension. These are not hardcoded values - you configure them in your collector configuration file.
|
<Tabs>
|
||||||
|
<Tab title="Pull-based Monitoring (Prometheus)">
|
||||||
|
This approach exposes metrics on port 9464 at the `/metrics` endpoint, allowing Prometheus to scrape the data. The metrics are exposed in Prometheus format but originate from OpenTelemetry instrumentation.
|
||||||
|
|
||||||
## Option 1: Pull-based Monitoring (Prometheus)
|
### Configuration
|
||||||
|
|
||||||
This approach exposes metrics on port 9464 at the `/metrics` endpoint, allowing Prometheus to scrape the data. The metrics are exposed in Prometheus format but originate from OpenTelemetry instrumentation.
|
1. **Enable Prometheus export in Infisical**:
|
||||||
|
|
||||||
### Configuration
|
```bash
|
||||||
|
OTEL_TELEMETRY_COLLECTION_ENABLED=true
|
||||||
|
OTEL_EXPORT_TYPE=prometheus
|
||||||
|
```
|
||||||
|
|
||||||
1. **Enable Prometheus export in Infisical**:
|
2. **Expose the metrics port** in your Infisical backend:
|
||||||
|
|
||||||
```bash
|
- **Docker**: Expose port 9464
|
||||||
OTEL_TELEMETRY_COLLECTION_ENABLED=true
|
- **Kubernetes**: Create a service exposing port 9464
|
||||||
OTEL_EXPORT_TYPE=prometheus
|
- **Other**: Ensure port 9464 is accessible to your monitoring stack
|
||||||
```
|
|
||||||
|
|
||||||
2. **Expose the metrics port** in your Infisical backend:
|
3. **Create Prometheus configuration** (`prometheus.yml`):
|
||||||
|
|
||||||
- **Docker**: Expose port 9464
|
```yaml
|
||||||
- **Kubernetes**: Create a service exposing port 9464
|
global:
|
||||||
- **Other**: Ensure port 9464 is accessible to your monitoring stack
|
scrape_interval: 30s
|
||||||
|
evaluation_interval: 30s
|
||||||
|
|
||||||
3. **Create Prometheus configuration** (`prometheus.yml`):
|
scrape_configs:
|
||||||
|
- job_name: "infisical"
|
||||||
|
scrape_interval: 30s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["infisical-backend:9464"] # Adjust hostname/port based on your deployment
|
||||||
|
metrics_path: "/metrics"
|
||||||
|
```
|
||||||
|
|
||||||
```yaml
|
**Note**: Replace `infisical-backend:9464` with the actual hostname and port where your Infisical backend is running. This could be:
|
||||||
global:
|
|
||||||
scrape_interval: 30s
|
|
||||||
evaluation_interval: 30s
|
|
||||||
|
|
||||||
scrape_configs:
|
- **Docker Compose**: `infisical-backend:9464` (service name)
|
||||||
- job_name: "infisical"
|
- **Kubernetes**: `infisical-backend.default.svc.cluster.local:9464` (service name)
|
||||||
scrape_interval: 30s
|
- **Bare Metal**: `192.168.1.100:9464` (actual IP address)
|
||||||
static_configs:
|
- **Cloud**: `your-infisical.example.com:9464` (domain name)
|
||||||
- targets: ["infisical-backend:9464"] # Adjust hostname/port based on your deployment
|
|
||||||
metrics_path: "/metrics"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Note**: Replace `infisical-backend:9464` with the actual hostname and port where your Infisical backend is running. This could be:
|
### Deployment Options
|
||||||
|
|
||||||
- **Docker Compose**: `infisical-backend:9464` (service name)
|
<Tabs>
|
||||||
- **Kubernetes**: `infisical-backend.default.svc.cluster.local:9464` (service name)
|
<Tab title="Docker Compose">
|
||||||
- **Bare Metal**: `192.168.1.100:9464` (actual IP address)
|
```yaml
|
||||||
- **Cloud**: `your-infisical.example.com:9464` (domain name)
|
services:
|
||||||
|
prometheus:
|
||||||
|
image: prom/prometheus:latest
|
||||||
|
ports:
|
||||||
|
- "9090:9090"
|
||||||
|
volumes:
|
||||||
|
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||||
|
command:
|
||||||
|
- "--config.file=/etc/prometheus/prometheus.yml"
|
||||||
|
|
||||||
### Deployment Options
|
grafana:
|
||||||
|
image: grafana/grafana:latest
|
||||||
|
ports:
|
||||||
|
- "3000:3000"
|
||||||
|
environment:
|
||||||
|
- GF_SECURITY_ADMIN_USER=admin
|
||||||
|
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
<Tab title="Kubernetes">
|
||||||
|
```yaml
|
||||||
|
# prometheus-deployment.yaml
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: prometheus
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: prometheus
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: prometheus
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: prometheus
|
||||||
|
image: prom/prometheus:latest
|
||||||
|
ports:
|
||||||
|
- containerPort: 9090
|
||||||
|
volumeMounts:
|
||||||
|
- name: config
|
||||||
|
mountPath: /etc/prometheus
|
||||||
|
volumes:
|
||||||
|
- name: config
|
||||||
|
configMap:
|
||||||
|
name: prometheus-config
|
||||||
|
|
||||||
#### Docker Compose
|
---
|
||||||
|
# prometheus-service.yaml
|
||||||
```yaml
|
apiVersion: v1
|
||||||
services:
|
kind: Service
|
||||||
prometheus:
|
metadata:
|
||||||
image: prom/prometheus:latest
|
name: prometheus
|
||||||
ports:
|
spec:
|
||||||
- "9090:9090"
|
selector:
|
||||||
volumes:
|
app: prometheus
|
||||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
|
||||||
command:
|
|
||||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
|
||||||
|
|
||||||
grafana:
|
|
||||||
image: grafana/grafana:latest
|
|
||||||
ports:
|
|
||||||
- "3000:3000"
|
|
||||||
environment:
|
|
||||||
- GF_SECURITY_ADMIN_USER=admin
|
|
||||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Kubernetes
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# prometheus-deployment.yaml
|
|
||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: prometheus
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app: prometheus
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app: prometheus
|
|
||||||
spec:
|
|
||||||
containers:
|
|
||||||
- name: prometheus
|
|
||||||
image: prom/prometheus:latest
|
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 9090
|
- port: 9090
|
||||||
volumeMounts:
|
targetPort: 9090
|
||||||
- name: config
|
type: ClusterIP
|
||||||
mountPath: /etc/prometheus
|
```
|
||||||
volumes:
|
</Tab>
|
||||||
- name: config
|
<Tab title="Helm">
|
||||||
configMap:
|
```bash
|
||||||
name: prometheus-config
|
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
||||||
|
helm install prometheus prometheus-community/prometheus \
|
||||||
|
--set server.config.global.scrape_interval=30s \
|
||||||
|
--set server.config.scrape_configs[0].job_name=infisical \
|
||||||
|
--set server.config.scrape_configs[0].static_configs[0].targets[0]=infisical-backend:9464
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
---
|
</Tab>
|
||||||
# prometheus-service.yaml
|
<Tab title="Push-based Monitoring (OTLP)">
|
||||||
apiVersion: v1
|
This approach sends metrics directly to an OpenTelemetry Collector via the OTLP protocol. This gives you the most flexibility as you can configure the collector to export to multiple backends simultaneously.
|
||||||
kind: Service
|
|
||||||
metadata:
|
|
||||||
name: prometheus
|
|
||||||
spec:
|
|
||||||
selector:
|
|
||||||
app: prometheus
|
|
||||||
ports:
|
|
||||||
- port: 9090
|
|
||||||
targetPort: 9090
|
|
||||||
type: ClusterIP
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Helm
|
### Configuration
|
||||||
|
|
||||||
```bash
|
1. **Enable OTLP export in Infisical**:
|
||||||
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
|
||||||
helm install prometheus prometheus-community/prometheus \
|
|
||||||
--set server.config.global.scrape_interval=30s \
|
|
||||||
--set server.config.scrape_configs[0].job_name=infisical \
|
|
||||||
--set server.config.scrape_configs[0].static_configs[0].targets[0]=infisical-backend:9464
|
|
||||||
```
|
|
||||||
|
|
||||||
## Option 2: Push-based Monitoring (OTLP)
|
```bash
|
||||||
|
OTEL_TELEMETRY_COLLECTION_ENABLED=true
|
||||||
|
OTEL_EXPORT_TYPE=otlp
|
||||||
|
OTEL_EXPORT_OTLP_ENDPOINT=http://otel-collector:4318/v1/metrics
|
||||||
|
OTEL_COLLECTOR_BASIC_AUTH_USERNAME=infisical
|
||||||
|
OTEL_COLLECTOR_BASIC_AUTH_PASSWORD=infisical
|
||||||
|
OTEL_OTLP_PUSH_INTERVAL=30000
|
||||||
|
```
|
||||||
|
|
||||||
This approach sends metrics directly to an OpenTelemetry Collector via the OTLP protocol. This gives you the most flexibility as you can configure the collector to export to multiple backends simultaneously.
|
2. **Create OpenTelemetry Collector configuration** (`otel-collector-config.yaml`):
|
||||||
|
|
||||||
### Configuration
|
```yaml
|
||||||
|
extensions:
|
||||||
|
health_check:
|
||||||
|
pprof:
|
||||||
|
zpages:
|
||||||
|
basicauth/server:
|
||||||
|
htpasswd:
|
||||||
|
inline: |
|
||||||
|
your_username:your_password
|
||||||
|
|
||||||
1. **Enable OTLP export in Infisical**:
|
receivers:
|
||||||
|
otlp:
|
||||||
|
protocols:
|
||||||
|
http:
|
||||||
|
endpoint: 0.0.0.0:4318
|
||||||
|
auth:
|
||||||
|
authenticator: basicauth/server
|
||||||
|
|
||||||
```bash
|
prometheus:
|
||||||
OTEL_TELEMETRY_COLLECTION_ENABLED=true
|
config:
|
||||||
OTEL_EXPORT_TYPE=otlp
|
scrape_configs:
|
||||||
OTEL_EXPORT_OTLP_ENDPOINT=http://otel-collector:4318/v1/metrics
|
- job_name: otel-collector
|
||||||
OTEL_COLLECTOR_BASIC_AUTH_USERNAME=infisical
|
scrape_interval: 30s
|
||||||
OTEL_COLLECTOR_BASIC_AUTH_PASSWORD=infisical
|
static_configs:
|
||||||
OTEL_OTLP_PUSH_INTERVAL=30000
|
- targets: [infisical-backend:9464]
|
||||||
```
|
metric_relabel_configs:
|
||||||
|
- action: labeldrop
|
||||||
|
regex: "service_instance_id|service_name"
|
||||||
|
|
||||||
2. **Create OpenTelemetry Collector configuration** (`otel-collector-config.yaml`):
|
processors:
|
||||||
|
batch:
|
||||||
|
|
||||||
```yaml
|
exporters:
|
||||||
extensions:
|
prometheus:
|
||||||
health_check:
|
endpoint: "0.0.0.0:8889"
|
||||||
pprof:
|
|
||||||
zpages:
|
|
||||||
basicauth/server:
|
|
||||||
htpasswd:
|
|
||||||
inline: |
|
|
||||||
your_username:your_password
|
|
||||||
|
|
||||||
receivers:
|
|
||||||
otlp:
|
|
||||||
protocols:
|
|
||||||
http:
|
|
||||||
endpoint: 0.0.0.0:4318
|
|
||||||
auth:
|
auth:
|
||||||
authenticator: basicauth/server
|
authenticator: basicauth/server
|
||||||
|
resource_to_telemetry_conversion:
|
||||||
|
enabled: true
|
||||||
|
|
||||||
prometheus:
|
service:
|
||||||
config:
|
extensions: [basicauth/server, health_check, pprof, zpages]
|
||||||
scrape_configs:
|
pipelines:
|
||||||
- job_name: otel-collector
|
metrics:
|
||||||
scrape_interval: 30s
|
receivers: [otlp]
|
||||||
static_configs:
|
processors: [batch]
|
||||||
- targets: [infisical-backend:9464]
|
exporters: [prometheus]
|
||||||
metric_relabel_configs:
|
```
|
||||||
- action: labeldrop
|
|
||||||
regex: "service_instance_id|service_name"
|
|
||||||
|
|
||||||
processors:
|
**Important**: Replace `your_username:your_password` with your chosen credentials. These must match the values you set in Infisical's `OTEL_COLLECTOR_BASIC_AUTH_USERNAME` and `OTEL_COLLECTOR_BASIC_AUTH_PASSWORD` environment variables.
|
||||||
batch:
|
|
||||||
|
|
||||||
exporters:
|
3. **Create Prometheus configuration** for the collector:
|
||||||
prometheus:
|
|
||||||
endpoint: "0.0.0.0:8889"
|
|
||||||
auth:
|
|
||||||
authenticator: basicauth/server
|
|
||||||
resource_to_telemetry_conversion:
|
|
||||||
enabled: true
|
|
||||||
|
|
||||||
service:
|
```yaml
|
||||||
extensions: [basicauth/server, health_check, pprof, zpages]
|
global:
|
||||||
pipelines:
|
scrape_interval: 30s
|
||||||
metrics:
|
evaluation_interval: 30s
|
||||||
receivers: [otlp]
|
|
||||||
processors: [batch]
|
|
||||||
exporters: [prometheus]
|
|
||||||
```
|
|
||||||
|
|
||||||
**Important**: Replace `your_username:your_password` with your chosen credentials. These must match the values you set in Infisical's `OTEL_COLLECTOR_BASIC_AUTH_USERNAME` and `OTEL_COLLECTOR_BASIC_AUTH_PASSWORD` environment variables.
|
scrape_configs:
|
||||||
|
- job_name: "otel-collector"
|
||||||
|
scrape_interval: 30s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["otel-collector:8889"] # Adjust hostname/port based on your deployment
|
||||||
|
metrics_path: "/metrics"
|
||||||
|
```
|
||||||
|
|
||||||
3. **Create Prometheus configuration** for the collector:
|
**Note**: Replace `otel-collector:8889` with the actual hostname and port where your OpenTelemetry Collector is running. This could be:
|
||||||
|
|
||||||
```yaml
|
- **Docker Compose**: `otel-collector:8889` (service name)
|
||||||
global:
|
- **Kubernetes**: `otel-collector.default.svc.cluster.local:8889` (service name)
|
||||||
scrape_interval: 30s
|
- **Bare Metal**: `192.168.1.100:8889` (actual IP address)
|
||||||
evaluation_interval: 30s
|
- **Cloud**: `your-collector.example.com:8889` (domain name)
|
||||||
|
|
||||||
scrape_configs:
|
### Deployment Options
|
||||||
- job_name: "otel-collector"
|
|
||||||
scrape_interval: 30s
|
|
||||||
static_configs:
|
|
||||||
- targets: ["otel-collector:8889"] # Adjust hostname/port based on your deployment
|
|
||||||
metrics_path: "/metrics"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Note**: Replace `otel-collector:8889` with the actual hostname and port where your OpenTelemetry Collector is running. This could be:
|
<Tabs>
|
||||||
|
<Tab title="Docker Compose">
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
otel-collector:
|
||||||
|
image: otel/opentelemetry-collector-contrib:latest
|
||||||
|
ports:
|
||||||
|
- 4318:4318 # OTLP http receiver
|
||||||
|
- 8889:8889 # Prometheus exporter metrics
|
||||||
|
volumes:
|
||||||
|
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
||||||
|
command:
|
||||||
|
- "--config=/etc/otelcol-contrib/config.yaml"
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
<Tab title="Kubernetes">
|
||||||
|
```yaml
|
||||||
|
# otel-collector-deployment.yaml
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: otel-collector
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: otel-collector
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: otel-collector
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: otel-collector
|
||||||
|
image: otel/opentelemetry-collector-contrib:latest
|
||||||
|
ports:
|
||||||
|
- containerPort: 4318
|
||||||
|
- containerPort: 8889
|
||||||
|
volumeMounts:
|
||||||
|
- name: config
|
||||||
|
mountPath: /etc/otelcol-contrib
|
||||||
|
volumes:
|
||||||
|
- name: config
|
||||||
|
configMap:
|
||||||
|
name: otel-collector-config
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
<Tab title="Helm">
|
||||||
|
```bash
|
||||||
|
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
|
||||||
|
helm install otel-collector open-telemetry/opentelemetry-collector \
|
||||||
|
--set config.receivers.otlp.protocols.http.endpoint=0.0.0.0:4318 \
|
||||||
|
--set config.exporters.prometheus.endpoint=0.0.0.0:8889
|
||||||
|
```
|
||||||
|
</Tab>
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
- **Docker Compose**: `otel-collector:8889` (service name)
|
</Tab>
|
||||||
- **Kubernetes**: `otel-collector.default.svc.cluster.local:8889` (service name)
|
</Tabs>
|
||||||
- **Bare Metal**: `192.168.1.100:8889` (actual IP address)
|
|
||||||
- **Cloud**: `your-collector.example.com:8889` (domain name)
|
|
||||||
|
|
||||||
### Deployment Options
|
|
||||||
|
|
||||||
#### Docker Compose
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
services:
|
|
||||||
otel-collector:
|
|
||||||
image: otel/opentelemetry-collector-contrib:latest
|
|
||||||
ports:
|
|
||||||
- 4318:4318 # OTLP http receiver
|
|
||||||
- 8889:8889 # Prometheus exporter metrics
|
|
||||||
volumes:
|
|
||||||
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
|
||||||
command:
|
|
||||||
- "--config=/etc/otelcol-contrib/config.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Kubernetes
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# otel-collector-deployment.yaml
|
|
||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: otel-collector
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app: otel-collector
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app: otel-collector
|
|
||||||
spec:
|
|
||||||
containers:
|
|
||||||
- name: otel-collector
|
|
||||||
image: otel/opentelemetry-collector-contrib:latest
|
|
||||||
ports:
|
|
||||||
- containerPort: 4318
|
|
||||||
- containerPort: 8889
|
|
||||||
volumeMounts:
|
|
||||||
- name: config
|
|
||||||
mountPath: /etc/otelcol-contrib
|
|
||||||
volumes:
|
|
||||||
- name: config
|
|
||||||
configMap:
|
|
||||||
name: otel-collector-config
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Helm
|
|
||||||
|
|
||||||
```bash
|
|
||||||
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
|
|
||||||
helm install otel-collector open-telemetry/opentelemetry-collector \
|
|
||||||
--set config.receivers.otlp.protocols.http.endpoint=0.0.0.0:4318 \
|
|
||||||
--set config.exporters.prometheus.endpoint=0.0.0.0:8889
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Metrics
|
## Available Metrics
|
||||||
|
|
||||||
@@ -327,154 +324,210 @@ Infisical exposes the following key metrics in OpenTelemetry format:
|
|||||||
|
|
||||||
These metrics track all HTTP API requests to Infisical, including request counts, latency, and errors. Use these to monitor overall API health, identify performance bottlenecks, and track usage patterns across users and machine identities.
|
These metrics track all HTTP API requests to Infisical, including request counts, latency, and errors. Use these to monitor overall API health, identify performance bottlenecks, and track usage patterns across users and machine identities.
|
||||||
|
|
||||||
#### Total API Requests
|
<AccordionGroup>
|
||||||
|
<Accordion title="Total API Requests">
|
||||||
|
**Metric Name**: `infisical.http.server.request.count`
|
||||||
|
|
||||||
- **Metric Name**: `infisical.http.server.request.count`
|
**Type**: Counter
|
||||||
- **Type**: Counter
|
|
||||||
- **Unit**: `{request}`
|
|
||||||
- **Description**: Total number of API requests to Infisical (covers both human users and machine identities)
|
|
||||||
- **Attributes**:
|
|
||||||
- `infisical.organization.id` (string): Organization ID
|
|
||||||
- `infisical.organization.name` (string): Organization name (e.g., "Platform Engineering Team")
|
|
||||||
- `infisical.user.id` (string, optional): User ID if human user
|
|
||||||
- `infisical.user.email` (string, optional): User email (e.g., "[email protected]")
|
|
||||||
- `infisical.identity.id` (string, optional): Machine identity ID
|
|
||||||
- `infisical.identity.name` (string, optional): Machine identity name (e.g., "prod-k8s-operator")
|
|
||||||
- `infisical.auth.method` (string, optional): Auth method used
|
|
||||||
- `http.request.method` (string): HTTP method (GET, POST, PUT, DELETE)
|
|
||||||
- `http.route` (string): API endpoint route pattern
|
|
||||||
- `http.response.status_code` (int): HTTP status code
|
|
||||||
- `infisical.project.id` (string, optional): Project ID
|
|
||||||
- `infisical.project.name` (string, optional): Project name
|
|
||||||
- `user_agent.original` (string, optional): User agent string
|
|
||||||
- `client.address` (string, optional): IP address
|
|
||||||
|
|
||||||
#### Request Duration
|
**Unit**: `{request}`
|
||||||
|
|
||||||
- **Metric Name**: `infisical.http.server.request.duration`
|
**Description**: Total number of API requests to Infisical (covers both human users and machine identities)
|
||||||
- **Type**: Histogram
|
|
||||||
- **Unit**: `s` (seconds)
|
|
||||||
- **Description**: API request latency
|
|
||||||
- **Buckets**: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
|
|
||||||
- **Attributes**:
|
|
||||||
- `infisical.organization.id` (string): Organization ID
|
|
||||||
- `infisical.organization.name` (string): Organization name
|
|
||||||
- `infisical.user.id` (string, optional): User ID if human user
|
|
||||||
- `infisical.user.email` (string, optional): User email
|
|
||||||
- `infisical.identity.id` (string, optional): Machine identity ID
|
|
||||||
- `infisical.identity.name` (string, optional): Machine identity name
|
|
||||||
- `http.request.method` (string): HTTP method
|
|
||||||
- `http.route` (string): API endpoint route pattern
|
|
||||||
- `http.response.status_code` (int): HTTP status code
|
|
||||||
- `infisical.project.id` (string, optional): Project ID
|
|
||||||
- `infisical.project.name` (string, optional): Project name
|
|
||||||
|
|
||||||
#### API Errors by Actor
|
**Attributes**:
|
||||||
|
- `infisical.organization.id` (string): Organization ID
|
||||||
|
- `infisical.organization.name` (string): Organization name (e.g., "Platform Engineering Team")
|
||||||
|
- `infisical.user.id` (string, optional): User ID if human user
|
||||||
|
- `infisical.user.email` (string, optional): User email (e.g., "[email protected]")
|
||||||
|
- `infisical.identity.id` (string, optional): Machine identity ID
|
||||||
|
- `infisical.identity.name` (string, optional): Machine identity name (e.g., "prod-k8s-operator")
|
||||||
|
- `infisical.auth.method` (string, optional): Auth method used
|
||||||
|
- `http.request.method` (string): HTTP method (GET, POST, PUT, DELETE)
|
||||||
|
- `http.route` (string): API endpoint route pattern
|
||||||
|
- `http.response.status_code` (int): HTTP status code
|
||||||
|
- `infisical.project.id` (string, optional): Project ID
|
||||||
|
- `infisical.project.name` (string, optional): Project name
|
||||||
|
- `user_agent.original` (string, optional): User agent string
|
||||||
|
- `client.address` (string, optional): IP address
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
- **Metric Name**: `infisical.http.server.error.count`
|
<Accordion title="Request Duration">
|
||||||
- **Type**: Counter
|
**Metric Name**: `infisical.http.server.request.duration`
|
||||||
- **Unit**: `{error}`
|
|
||||||
- **Description**: API errors grouped by actor (for identifying misconfigured services)
|
**Type**: Histogram
|
||||||
- **Attributes**:
|
|
||||||
- `infisical.organization.id` (string): Organization ID
|
**Unit**: `s` (seconds)
|
||||||
- `infisical.organization.name` (string): Organization name
|
|
||||||
- `infisical.user.id` (string, optional): User ID if human
|
**Description**: API request latency
|
||||||
- `infisical.user.email` (string, optional): User email
|
|
||||||
- `infisical.identity.id` (string, optional): Identity ID if machine
|
**Buckets**: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
|
||||||
- `infisical.identity.name` (string, optional): Identity name
|
|
||||||
- `http.route` (string): API endpoint where error occurred
|
**Attributes**:
|
||||||
- `http.request.method` (string): HTTP method
|
- `infisical.organization.id` (string): Organization ID
|
||||||
- `error.type` (string): Error category/type (client_error, server_error, auth_error, rate_limit_error, etc.)
|
- `infisical.organization.name` (string): Organization name
|
||||||
- `infisical.project.id` (string, optional): Project ID
|
- `infisical.user.id` (string, optional): User ID if human user
|
||||||
- `infisical.project.name` (string, optional): Project name
|
- `infisical.user.email` (string, optional): User email
|
||||||
- `client.address` (string, optional): IP address
|
- `infisical.identity.id` (string, optional): Machine identity ID
|
||||||
- `user_agent.original` (string, optional): User agent information
|
- `infisical.identity.name` (string, optional): Machine identity name
|
||||||
|
- `http.request.method` (string): HTTP method
|
||||||
|
- `http.route` (string): API endpoint route pattern
|
||||||
|
- `http.response.status_code` (int): HTTP status code
|
||||||
|
- `infisical.project.id` (string, optional): Project ID
|
||||||
|
- `infisical.project.name` (string, optional): Project name
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="API Errors by Actor">
|
||||||
|
**Metric Name**: `infisical.http.server.error.count`
|
||||||
|
|
||||||
|
**Type**: Counter
|
||||||
|
|
||||||
|
**Unit**: `{error}`
|
||||||
|
|
||||||
|
**Description**: API errors grouped by actor (for identifying misconfigured services)
|
||||||
|
|
||||||
|
**Attributes**:
|
||||||
|
- `infisical.organization.id` (string): Organization ID
|
||||||
|
- `infisical.organization.name` (string): Organization name
|
||||||
|
- `infisical.user.id` (string, optional): User ID if human
|
||||||
|
- `infisical.user.email` (string, optional): User email
|
||||||
|
- `infisical.identity.id` (string, optional): Identity ID if machine
|
||||||
|
- `infisical.identity.name` (string, optional): Identity name
|
||||||
|
- `http.route` (string): API endpoint where error occurred
|
||||||
|
- `http.request.method` (string): HTTP method
|
||||||
|
- `error.type` (string): Error category/type (client_error, server_error, auth_error, rate_limit_error, etc.)
|
||||||
|
- `infisical.project.id` (string, optional): Project ID
|
||||||
|
- `infisical.project.name` (string, optional): Project name
|
||||||
|
- `client.address` (string, optional): IP address
|
||||||
|
- `user_agent.original` (string, optional): User agent information
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
### Secret Operations Metrics
|
### Secret Operations Metrics
|
||||||
|
|
||||||
These metrics provide visibility into secret access patterns, helping you understand which secrets are being accessed, by whom, and from where. Essential for security auditing and access pattern analysis.
|
These metrics provide visibility into secret access patterns, helping you understand which secrets are being accessed, by whom, and from where. Essential for security auditing and access pattern analysis.
|
||||||
|
|
||||||
#### Secret Read Operations
|
<AccordionGroup>
|
||||||
|
<Accordion title="Secret Read Operations">
|
||||||
- **Metric Name**: `infisical.secret.read.count`
|
**Metric Name**: `infisical.secret.read.count`
|
||||||
- **Type**: Counter
|
|
||||||
- **Unit**: `{operation}`
|
**Type**: Counter
|
||||||
- **Description**: Number of secret read operations
|
|
||||||
- **Attributes**:
|
**Unit**: `{operation}`
|
||||||
- `infisical.organization.id` (string): Organization ID
|
|
||||||
- `infisical.organization.name` (string): Organization name
|
**Description**: Number of secret read operations
|
||||||
- `infisical.project.id` (string): Project ID
|
|
||||||
- `infisical.project.name` (string): Project name (e.g., "payment-service-secrets")
|
**Attributes**:
|
||||||
- `infisical.environment` (string): Environment (dev, staging, prod)
|
- `infisical.organization.id` (string): Organization ID
|
||||||
- `infisical.secret.path` (string): Path to secrets (e.g., "/microservice-a/database")
|
- `infisical.organization.name` (string): Organization name
|
||||||
- `infisical.secret.name` (string, optional): Name of secret
|
- `infisical.project.id` (string): Project ID
|
||||||
- `infisical.user.id` (string, optional): User ID if human
|
- `infisical.project.name` (string): Project name (e.g., "payment-service-secrets")
|
||||||
- `infisical.user.email` (string, optional): User email
|
- `infisical.environment` (string): Environment (dev, staging, prod)
|
||||||
- `infisical.identity.id` (string, optional): Machine identity ID
|
- `infisical.secret.path` (string): Path to secrets (e.g., "/microservice-a/database")
|
||||||
- `infisical.identity.name` (string, optional): Machine identity name
|
- `infisical.secret.name` (string, optional): Name of secret
|
||||||
- `user_agent.original` (string, optional): User agent/SDK information
|
- `infisical.user.id` (string, optional): User ID if human
|
||||||
- `client.address` (string, optional): IP address
|
- `infisical.user.email` (string, optional): User email
|
||||||
|
- `infisical.identity.id` (string, optional): Machine identity ID
|
||||||
|
- `infisical.identity.name` (string, optional): Machine identity name
|
||||||
|
- `user_agent.original` (string, optional): User agent/SDK information
|
||||||
|
- `client.address` (string, optional): IP address
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
### Authentication Metrics
|
### Authentication Metrics
|
||||||
|
|
||||||
These metrics track authentication attempts and outcomes, enabling you to monitor login success rates, detect potential security threats, and identify authentication issues.
|
These metrics track authentication attempts and outcomes, enabling you to monitor login success rates, detect potential security threats, and identify authentication issues.
|
||||||
|
|
||||||
#### Login Attempts
|
<AccordionGroup>
|
||||||
|
<Accordion title="Login Attempts">
|
||||||
- **Metric Name**: `infisical.auth.attempt.count`
|
**Metric Name**: `infisical.auth.attempt.count`
|
||||||
- **Type**: Counter
|
|
||||||
- **Unit**: `{attempt}`
|
**Type**: Counter
|
||||||
- **Description**: Authentication attempts (both successful and failed)
|
|
||||||
- **Attributes**:
|
**Unit**: `{attempt}`
|
||||||
- `infisical.organization.id` (string): Organization ID
|
|
||||||
- `infisical.organization.name` (string): Organization name
|
**Description**: Authentication attempts (both successful and failed)
|
||||||
- `infisical.user.id` (string, optional): User ID if human (if identifiable)
|
|
||||||
- `infisical.user.email` (string, optional): User email (if identifiable)
|
**Attributes**:
|
||||||
- `infisical.identity.id` (string, optional): Identity ID if machine (if identifiable)
|
- `infisical.organization.id` (string): Organization ID
|
||||||
- `infisical.identity.name` (string, optional): Identity name (if identifiable)
|
- `infisical.organization.name` (string): Organization name
|
||||||
- `infisical.auth.method` (string): Authentication method attempted
|
- `infisical.user.id` (string, optional): User ID if human (if identifiable)
|
||||||
- `infisical.auth.result` (string): success or failure
|
- `infisical.user.email` (string, optional): User email (if identifiable)
|
||||||
- `error.type` (string, optional): Reason for failure if failed (invalid_credentials, expired_token, invalid_token, etc.)
|
- `infisical.identity.id` (string, optional): Identity ID if machine (if identifiable)
|
||||||
- `client.address` (string): IP address
|
- `infisical.identity.name` (string, optional): Identity name (if identifiable)
|
||||||
- `user_agent.original` (string, optional): User agent/client information
|
- `infisical.auth.method` (string): Authentication method attempted
|
||||||
- `infisical.auth.attempt.username` (string, optional): Attempted username/email (if available)
|
- `infisical.auth.result` (string): success or failure
|
||||||
|
- `error.type` (string, optional): Reason for failure if failed (invalid_credentials, expired_token, invalid_token, etc.)
|
||||||
|
- `client.address` (string): IP address
|
||||||
|
- `user_agent.original` (string, optional): User agent/client information
|
||||||
|
- `infisical.auth.attempt.username` (string, optional): Attempted username/email (if available)
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
### Legacy Metrics
|
### Legacy Metrics
|
||||||
|
|
||||||
These metrics are from the previous instrumentation and may be deprecated in future versions. Consider migrating to the new Core API Metrics for more comprehensive observability.
|
These metrics are from the previous instrumentation and may be deprecated in future versions. Consider migrating to the new Core API Metrics for more comprehensive observability.
|
||||||
|
|
||||||
- `API_latency` - API request latency histogram in milliseconds (Labels: `route`, `method`, `statusCode`)
|
<AccordionGroup>
|
||||||
- `API_errors` - API error count histogram (Labels: `route`, `method`, `type`, `name`)
|
<Accordion title="API_latency">
|
||||||
|
API request latency histogram in milliseconds
|
||||||
|
|
||||||
|
- **Labels**: `route`, `method`, `statusCode`
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="API_errors">
|
||||||
|
API error count histogram
|
||||||
|
|
||||||
|
- **Labels**: `route`, `method`, `type`, `name`
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
### Integration & Secret Sync Metrics
|
### Integration & Secret Sync Metrics
|
||||||
|
|
||||||
These metrics monitor secret synchronization operations between Infisical and external systems, helping you track sync health, identify integration failures, and troubleshoot connectivity issues.
|
These metrics monitor secret synchronization operations between Infisical and external systems, helping you track sync health, identify integration failures, and troubleshoot connectivity issues.
|
||||||
|
|
||||||
- `integration_secret_sync_errors` - Integration secret sync error count
|
<AccordionGroup>
|
||||||
|
<Accordion title="integration_secret_sync_errors">
|
||||||
|
Integration secret sync error count
|
||||||
|
|
||||||
- **Labels**: `version`, `integration`, `integrationId`, `type`, `status`, `name`, `projectId`
|
- **Labels**: `version`, `integration`, `integrationId`, `type`, `status`, `name`, `projectId`
|
||||||
- **Example**: Monitor integration sync failures across different services
|
- **Example**: Monitor integration sync failures across different services
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
- `secret_sync_sync_secrets_errors` - Secret sync operation error count
|
<Accordion title="secret_sync_sync_secrets_errors">
|
||||||
|
Secret sync operation error count
|
||||||
|
|
||||||
- **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name`
|
- **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name`
|
||||||
- **Example**: Track secret sync failures to external systems
|
- **Example**: Track secret sync failures to external systems
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
- `secret_sync_import_secrets_errors` - Secret import operation error count
|
<Accordion title="secret_sync_import_secrets_errors">
|
||||||
|
Secret import operation error count
|
||||||
|
|
||||||
- **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name`
|
- **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name`
|
||||||
- **Example**: Monitor secret import failures
|
- **Example**: Monitor secret import failures
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
- `secret_sync_remove_secrets_errors` - Secret removal operation error count
|
<Accordion title="secret_sync_remove_secrets_errors">
|
||||||
- **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name`
|
Secret removal operation error count
|
||||||
- **Example**: Track secret removal operation failures
|
|
||||||
|
- **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name`
|
||||||
|
- **Example**: Track secret removal operation failures
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
### System Metrics
|
### System Metrics
|
||||||
|
|
||||||
These low-level HTTP metrics are automatically collected by OpenTelemetry's instrumentation layer, providing baseline performance data for all HTTP traffic.
|
These low-level HTTP metrics are automatically collected by OpenTelemetry's instrumentation layer, providing baseline performance data for all HTTP traffic.
|
||||||
|
|
||||||
- `http_server_duration` - HTTP server request duration metrics (histogram buckets, count, sum)
|
<AccordionGroup>
|
||||||
- `http_client_duration` - HTTP client request duration metrics (histogram buckets, count, sum)
|
<Accordion title="http_server_duration">
|
||||||
|
HTTP server request duration metrics (histogram buckets, count, sum)
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="http_client_duration">
|
||||||
|
HTTP client request duration metrics (histogram buckets, count, sum)
|
||||||
|
</Accordion>
|
||||||
|
</AccordionGroup>
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user