README

Utopia framework usage library is a simple and lite library for managing application usage statistics. This library is aiming to be as simple and easy to learn and use. This library is maintained by the Appwrite team.

Although this library is part of the Utopia Framework project it is dependency free and can be used as standalone with any other PHP project or framework.

Features

• Two Table Architecture: Separate events and gauges tables optimized for their access patterns
• Events Table: Request-level metrics with dedicated columns for every dimension we filter on (path/method/status, service/resource ids, team ids, country/region, hostname, parsed UA fields)
• Gauges Table: Simple resource snapshots (storage size, user count, etc.)
• Query-Time Aggregation: No write-time period fan-out — aggregate by any interval at query time
• Daily Materialized View: Pre-aggregated daily SummingMergeTree for fast billing queries
• Pluggable Adapters: ClickHouse (production) and Database (development/testing)
• In-Memory Buffering: Collect metrics and flush in batch for high-throughput scenarios
• Rich Queries: Filter, sort, paginate using Utopia\Query\Query objects
• Multi-Tenant: Shared tables with tenant isolation via string tenant IDs
• LowCardinality Columns: Country uses LowCardinality(String) for efficient storage
• Bloom Filter Indexes: Fast filtering on all event columns

Getting Started

Install using composer:

composer require utopia-php/usage

Using ClickHouse Adapter

<?php

use Utopia\Usage\Usage;
use Utopia\Usage\Adapter\ClickHouse;

// Configuration is fixed at construction. The adapter is fully stateless —
// the tenant is passed explicitly on every call (see below).
$adapter = new ClickHouse(
    host: 'clickhouse-server',
    username: 'default',
    password: '',
    port: 8123,
    secure: false,
    namespace: 'my_app',
    sharedTables: true,
);

$usage = new Usage($adapter);
$usage->setup(); // Creates events, gauges, and daily MV tables

Using Database Adapter

<?php

use Utopia\Usage\Usage;
use Utopia\Usage\Adapter\Database as DatabaseAdapter;

$adapter = new DatabaseAdapter($database); // Utopia\Database\Database instance
$usage = new Usage($adapter);
$usage->setup();

Multi-tenancy

Usage is stateless: every query/mutation takes the tenant as its first argument, and addBatch carries a tenant on each metric row (so one batch can span tenants). This makes a single Usage instance safe to share across tenants and coroutines.

$usage->getTotal('project_123', 'bandwidth');           // read for one tenant
$usage->addBatch([
    ['tenant' => 'project_123', 'metric' => 'bandwidth', 'value' => 5000, 'tags' => []],
    ['tenant' => 'project_456', 'metric' => 'bandwidth', 'value' => 2000, 'tags' => []],
], Usage::TYPE_EVENT);

Callers that only ever touch one tenant can bind it once with the Tenant decorator, which forwards to Usage with the tenant pre-filled (and stamps it onto every addBatch row):

use Utopia\Usage\Tenant;

$tenant = new Tenant($usage, 'project_123');
$tenant->getTotal('bandwidth');                          // no tenant argument
$tenant->addBatch([
    ['metric' => 'bandwidth', 'value' => 5000, 'tags' => []], // tenant stamped automatically
], Usage::TYPE_EVENT);

Metric Types

Events (Additive)

Events are request-level metrics like bandwidth, executions, API calls. They are summed when aggregated.

Event-specific columns (see Metric::EVENT_COLUMNS): path, method, status, service, resource, resourceId, resourceInternalId, teamId, teamInternalId, country, region, hostname, osCode, osName, osVersion, clientType, clientCode, clientName, clientVersion, clientEngine, clientEngineVersion, deviceName, deviceBrand, deviceModel.

use Utopia\Usage\Accumulator;

// Buffer metrics in memory and flush them in batch
$accumulator = new Accumulator($usage);

// Collect events — tenant first; values accumulate in-memory (summed per tenant+metric)
$accumulator->collect('project_123', 'bandwidth', 5000, Usage::TYPE_EVENT, [
    'path' => '/v1/storage/files',
    'method' => 'POST',
    'status' => '201',
    'service' => 'storage',
    'resource' => 'bucket',
    'resourceId' => 'abc123',
    'resourceInternalId' => '42',
    'teamId' => 'team_x',
    'teamInternalId' => '7',
    'country' => 'US',
    'region' => 'us-east',
    'hostname' => 'app.example.com',
    'osName' => 'iOS',
    'clientName' => 'Appwrite SDK',
    'deviceName' => 'smartphone',
]);

Strict tag keys

All keys passed in the tags array must map to a known event or gauge column (see Metric::EVENT_COLUMNS and Metric::GAUGE_COLUMNS). Unknown keys throw at write time — there is no JSON catch-all. To add a new dimension, widen the schema and bump the library version.

Gauges (Point-in-Time)

Gauges are resource snapshots like storage size, user count, file count. Last-write-wins semantics.

Gauge-specific columns (see Metric::GAUGE_COLUMNS): teamId, teamInternalId, resourceId, resourceInternalId.

// Collect gauges — last value wins per tenant+metric in buffer
$accumulator->collect('project_123', 'users', 1500, Usage::TYPE_GAUGE);
$accumulator->collect('project_123', 'storage.size', 1048576, Usage::TYPE_GAUGE, [
    'teamId' => 'team_x',
    'teamInternalId' => '7',
    'resourceId' => 'abc123',
    'resourceInternalId' => '42',
]);

Flushing

The accumulator exposes raw signals — count() (buffered entries) and elapsedSeconds() (seconds since last flush) — and leaves the flush policy to the caller.

// Flush when the buffer grows large or enough time has passed
if ($accumulator->count() >= 5000 || $accumulator->elapsedSeconds() >= 10) {
    $accumulator->flush(); // Writes events to events table, gauges to gauges table
}

Batch Writes

// Write directly without buffering — each row carries its tenant
$usage->addBatch([
    ['tenant' => 'project_123', 'metric' => 'requests', 'value' => 100, 'tags' => ['path' => '/v1/users', 'method' => 'GET']],
    ['tenant' => 'project_123', 'metric' => 'bandwidth', 'value' => 50000, 'tags' => ['country' => 'DE', 'region' => 'fra']],
], Usage::TYPE_EVENT);

$usage->addBatch([
    ['tenant' => 'project_123', 'metric' => 'users', 'value' => 42, 'tags' => ['teamId' => 'team_x']],
], Usage::TYPE_GAUGE);

Querying Metrics

Find with Query Objects

use Utopia\Query\Query;

// Find events filtered by metric and time range (tenant first)
$metrics = $usage->find('project_123', [
    Query::equal('metric', ['bandwidth']),
    Query::equal('country', ['US']),
    Query::greaterThanEqual('time', '2026-01-01'),
    Query::orderDesc('time'),
    Query::limit(100),
], Usage::TYPE_EVENT);

// Find gauges
$gauges = $usage->find('project_123', [
    Query::equal('metric', ['users', 'storage.size']),
], Usage::TYPE_GAUGE);

// Query both tables (type = null)
$all = $usage->find('project_123', [
    Query::equal('metric', ['bandwidth']),
]);

Totals

// Get total for a single metric (SUM for events, latest for gauges)
$total = $usage->getTotal('project_123', 'bandwidth', [
    Query::greaterThanEqual('time', '2026-03-01'),
], Usage::TYPE_EVENT);

// Batch totals — single query with GROUP BY
$totals = $usage->getTotalBatch(
    'project_123',
    ['bandwidth', 'executions', 'requests'],
    [Query::greaterThanEqual('time', '2026-03-01')],
    Usage::TYPE_EVENT
);

Time Series

// Get time series with query-time aggregation
// Events: SUM per bucket, Gauges: argMax per bucket
$series = $usage->getTimeSeries(
    tenant: 'project_123',
    metrics: ['bandwidth', 'requests'],
    interval: '1d',        // '1h' or '1d'
    startDate: '2026-03-01',
    endDate: '2026-04-01',
    zeroFill: true,        // Fill gaps with zeros
    type: Usage::TYPE_EVENT
);

// Returns: ['bandwidth' => ['total' => 5000000, 'data' => [['value' => 100, 'date' => '...'], ...]]]

Billing Queries (Daily MV)

The daily materialized view pre-aggregates events by metric + tenant + day for fast billing:

// Sum a single metric from the daily table
$total = $usage->sumDaily('project_123', [
    Query::equal('metric', ['bandwidth']),
    Query::between('time', '2026-03-01', '2026-04-01'),
]);

// Sum multiple metrics in one query
$totals = $usage->sumDailyBatch(
    'project_123',
    ['bandwidth', 'executions', 'storage.size'],
    [Query::between('time', '2026-03-01', '2026-04-01')]
);
// Returns: ['bandwidth' => 5000000, 'executions' => 12345, 'storage.size' => 0]

// Find daily aggregated rows
$rows = $usage->findDaily('project_123', [
    Query::equal('metric', ['bandwidth']),
    Query::orderDesc('time'),
    Query::limit(30),
]);

Purge

// Purge all event metrics older than 90 days
$usage->purge('project_123', [
    Query::lessThan('time', '2026-01-01'),
], Usage::TYPE_EVENT);

// Purge all gauge metrics
$usage->purge('project_123', [], Usage::TYPE_GAUGE);

Architecture

ClickHouse Tables

Events Table Schema

Gauges Table Schema

Daily Table Schema

Creating Custom Adapters

Extend Utopia\Usage\Adapter and implement:

• getName(), setup(), healthCheck()
• addBatch(array $metrics, string $type, int $batchSize): bool (each metric carries its own tenant)
• find(string $tenant, array $queries, ?string $type): array
• count(string $tenant, array $queries, ?string $type, ?int $max): int
• sum(string $tenant, array $queries, string $attribute, string $type): int
• getTimeSeries(string $tenant, array $metrics, string $interval, string $startDate, string $endDate, array $queries, bool $zeroFill, ?string $type): array
• getTotal(string $tenant, string $metric, array $queries, ?string $type): int
• getTotalBatch(string $tenant, array $metrics, array $queries, ?string $type): array
• findDaily(string $tenant, array $queries): array
• sumDaily(string $tenant, array $queries, string $attribute): int
• sumDailyBatch(string $tenant, array $metrics, array $queries): array
• purge(string $tenant, array $queries, ?string $type): bool

All configuration — namespace, database, shared-tables mode, async inserts, query logging — is set once via the adapter constructor (see "Using ClickHouse Adapter" above). Adapters hold no per-request state; the tenant is passed explicitly on every call, so one instance is safe to share across tenants and coroutines.

System Requirements

Utopia Framework requires PHP 8.0 or later. We recommend using the latest PHP version whenever possible.

Copyright and license

The MIT License (MIT) http://www.opensource.org/licenses/mit-license.php