Prometheus Observability

Enterprise exposes a Prometheus-compatible metrics endpoint for production monitoring. The endpoint serves metrics in the standard Prometheus exposition format, ready for scraping by Prometheus, Grafana Agent, or any compatible collector.

Metrics endpoint

GET /api/metrics

The endpoint requires authentication. The caller must hold meta_read_access on the system store (SystemDatabase/system) through a capability assigned to their own user — capabilities held by organizations or teams the user belongs to do not count on their own; the grant must sit on the user.

• The built-in admin user already satisfies the check via its Capability/server_access grant on Organization/admin, which subsumes the system store.
• Requests without the capability receive 403 — including credential-less requests, which authenticate as User/anonymous.
• Malformed credentials receive 401.

curl -s -u admin:root http://localhost:6363/api/metrics

Granting a scrape account

For production monitoring, prefer a dedicated user over the admin account. The narrowest grant is a custom role carrying only meta_read_access, assigned on the admin organization (the scope that contains the system store). Note that an organization-scoped grant also covers the databases under that organization — capability scope checks follow the resource hierarchy.

# 1. Create the scrape user
curl -s -u admin:root -X POST http://localhost:6363/api/users \
  -H "Content-Type: application/json" \
  -d '{"name": "metrics-scraper", "password": "change-me"}'

# 2. Create a role with only meta_read_access
curl -s -u admin:root -X POST http://localhost:6363/api/roles \
  -H "Content-Type: application/json" \
  -d '{"name": "metrics-reader", "action": ["meta_read_access"]}'

# 3. Grant the role to the user on the admin organization
curl -s -u admin:root -X POST http://localhost:6363/api/capabilities \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "grant",
    "scope_type": "organization",
    "scope": "admin",
    "user": "metrics-scraper",
    "roles": ["metrics-reader"]
  }'

See Capabilities API modes for the grant/revoke payload reference.

Example response

# HELP build_info Server version and build information
# TYPE build_info gauge
build_info{version="12.0.7",git_hash="abc1234",store_version="0.19.8",edition="enterprise"} 1
# HELP uptime_seconds Time since the server process started
# TYPE uptime_seconds gauge
uptime_seconds 3842.7
# HELP atoms Number of atoms in the Prolog atom table
# TYPE atoms gauge
atoms 29510
# HELP layer_cache_entries_total Total entries in the layer object cache
# TYPE layer_cache_entries_total gauge
layer_cache_entries_total 42
# HELP layer_cache_entries_live Live (in use) layer cache entries
# TYPE layer_cache_entries_live gauge
layer_cache_entries_live 38
# HELP layer_cache_entries_dead Dead (evicted, not yet freed) layer cache entries
# TYPE layer_cache_entries_dead gauge
layer_cache_entries_dead 4
# HELP worker_pool_busy Number of worker threads currently processing requests
# TYPE worker_pool_busy gauge
worker_pool_busy 2
# HELP lru_cache_limit_bytes Configured LRU archive cache limit
# TYPE lru_cache_limit_bytes gauge
lru_cache_limit_bytes 536870912
# HELP lru_cache_used_bytes Current LRU archive cache usage
# TYPE lru_cache_used_bytes gauge
lru_cache_used_bytes 8388608

Metric names have no prefix by default; a prefix can be configured in the enterprise config (metrics_prefix/1).

Available metrics

Server information

Memory and stack

Code and atoms

Growth in these metrics without matching traffic indicates retained data — asserted clauses pinning atoms, for example.

Worker pool and threads

Foreign blobs

Each foreign blob atom can pin native memory (buffers, layer handles) that is invisible to the Prolog space metrics. Growth in a blob type without matching traffic means retained references.

Layer cache

The layer cache stores graph layers in memory for fast access. These metrics help you understand cache pressure and tune the LRU eviction policy.

LRU cache

The LRU (Least Recently Used) cache manages eviction of graph layers when memory pressure increases.

Prometheus scrape configuration

Add TerminusDB to your Prometheus scrape_configs. The endpoint requires authentication, so provide credentials for a user holding meta_read_access on the system store:

scrape_configs:
  - job_name: 'terminusdb'
    scrape_interval: 15s
    metrics_path: '/api/metrics'
    basic_auth:
      username: metrics-scraper
      password_file: /etc/prometheus/secrets/terminusdb-metrics-password
    static_configs:
      - targets: ['localhost:6363']

Grafana dashboard tips

Useful panels for a TerminusDB Grafana dashboard:

• Uptime — uptime_seconds
• Cache pressure — layer_cache_entries_dead climbing relative to layer_cache_entries_total
• Memory pressure — lru_cache_used_bytes / lru_cache_limit_bytes
• Pool saturation — worker_pool_busy against worker_pool_total, plus worker_busy_seconds for stuck requests
• Atom growth — atoms alongside atom_gc_atoms_reclaimed_total to distinguish reachable data from an AGC backlog
• Active requests — http_requests_active and http_connections_active for concurrency monitoring
• Cache eviction headroom — lru_cache_limit_bytes - lru_cache_used_bytes

Further reading

• Configuration Reference — LRU cache tuning variables
• Capabilities API modes — granting roles and capabilities
• Enterprise Overview — all enterprise features