README

A Symfony bundle to interact with storage (via Flysystem). It exposes commands, controllers and twig utilities. All of the underlying storage happens through Flysystem.

Used in production by

• kpa
• md
• mediary
• showcase
• zm

(derived from showcase's Component/Site scan — apps whose composer.json requires survos/storage-bundle and have a deployed dokku site)

Quickstart

symfony new storage-demo --webapp && cd storage-demo
composer require survos/storage-bundle

Optional thumbnail support:

composer require survos/imgproxy-bundle

Without imgproxy, the bundle still works normally and falls back to direct object links instead of generated image thumbnails/previews.

Configure Flysystem, including the relevant env vars if using something besides local

# config/packages/flysystem.yaml
# Read the documentation at https://github.com/thephpleague/flysystem-bundle/blob/master/docs/1-getting-started.md
flysystem:
    storages:
        default.storage:
            adapter: 'aws'
            # visibility: public # Make the uploaded file publicly accessible in S3
            options:
                client: 'Aws\S3\S3Client' # The service ID of the Aws\S3\S3Client instance
                bucket: '%env(AWS_S3_BUCKET_NAME)%'
                streamReads: true
                prefix: '%env(S3_STORAGE_PREFIX)%'
when@dev:
    flysystem:
        storages:
            default.storage:
                adapter: 'local'
                options:
                    directory: '%kernel.project_dir%/public/storage'

symfony new storage-demo --webapp && cd storage-demo
composer require survos/storage-bundle
bin/console storage:config
bin/console storage:list

You can browse interactively with the basic admin controller.

composer require survos/simple-datatables-bundle
symfony server:start -d
symfony open:local --path=/storage/zones

Storage zones (and their credentials) are configured in flysystem.yaml as shown above — local during dev, S3/Bunny/etc. in production via env vars.

To see what got wired at runtime, run the read-only diagnostic:

bin/console storage:config

It dumps the resolved zones and their adapters (class, root/bucket), which is handy for confirming env vars resolved correctly. It makes no changes and needs no credentials.

Legacy note: earlier versions shipped a Bunny-specific storage:config <api-key> that fetched per-zone keys from the Bunny dashboard. The bundle is now backend-agnostic (Flysystem), so that key-fetcher is gone; storage:config is purely diagnostic.

Your application now has a bare-bones controller located at /admin/storage, you may want to secure this route in security.yaml, or configure it in config/routes/survos_storage.yaml.

You also have access to a command line interface.

bin/console storage:list 

+------------- museado/ -----+--------+
| ObjectName     | Path      | Length |
+----------------+-----------+--------+
| photos finales | /museado/ | 0      |
+----------------+-----------+--------+

Live browsing and persisted inventory

Flysystem browsing lists the current backend directly. Doctrine inventory stores StorageNode records with parent/child relationships; tree-bundle is required for hierarchy support. Browsing a zone does not require populating its inventory.

Populate through the StorageNode workflow

Flysystem remains authoritative. Doctrine stores a disposable inventory. A StorageNode is either a directory or a file; sidecars are ordinary files.

survos_storage:
    tracking_lock_factory: lock.workflow.factory
    storages:
        archive.storage:
            track: true
            sidecar_prefix: o
            content_patterns: ['o/*.json']

bin/console storage:populate archive.storage o
# Run consumers only when ready to process the queued scan:
bin/console messenger:consume storage.fetch

Use a shared workflow lock store and configure the storage.fetch transport. --concurrency needs Symfony Messenger's amphp/parallel support; otherwise run multiple normal workers. sync must also be configured for explicit --sync use.

Scans default to listing-only. Each directory transition performs one shallow Flysystem listing, records the metadata supplied by the adapter, and queues child directories recursively. Files finish at StorageNode::STATUS_FILE_LISTED (file_listed); they are not waiting for processing. The scanner does not queue per-file existence, size or modification-time requests, download file contents, or inspect archive members. fileCount counts direct files in a storage directory, not files inside ZIPs or bags. Listing metadata can be absent when the adapter does not supply it. Application discovery callbacks may explicitly do extra work.

To enable per-file checks and configured content caching:

bin/console storage:populate archive.storage o --load-files

With --load-files, newly discovered files use the load_file transition. It checks existence and refreshes metadata; paths matching content_patterns are read and cached in meta[StorageNode::CONTENT], with SHA-256 in meta[StorageNode::CONTENT_SHA256]. It emits StorageFileLoadedEvent for application enrichment, then finishes at file_loaded. The flag does not itself enable media analysis or archive inspection. The same flag works with explicit --sync.

On rescans, directories are revisited. In listing-only mode, encountered files become file_listed without refreshing previously cached contents. With --load-files, files are queued when listing metadata changed, configured contents are missing, or loading is unfinished. Unseen/deleted nodes are not purged by a listing. Cached content from earlier scans must not be treated as freshly verified merely because a node was listed again.

Recorded scan request

DirectoryScanStarter::start($zone, $path, sync: false, loadFiles: false) records meta[StorageNode::SCAN_REQUEST] on the root and copies it to discovered children:

• id: a new request identifier for each start;
• requestedAt: ISO 8601 request time;
• root: the normalized relative root path;
• recursive: true (traversal is always recursive);
• loadFiles: the requested per-file mode.

Each successfully listed directory also records meta[StorageNode::SCANNED_AT]. The command prints the mode and request ID. These fields describe the latest scan observed by each node, not an append-only run history or a whole-tree completion timestamp. sync is an execution scope, not a persisted scan option. Roots created directly without the starter default to listing-only and have no generated request ID/time. Avoid overlapping scans of the same tree with different modes: the request metadata lives on the shared nodes.

Compatibility and application callbacks

This changes the default for all consuming applications. Callers that relied on population to warm sidecar caches must add --load-files, or pass loadFiles: true to DirectoryScanStarter::start(). content_patterns alone does not opt a scan in. --dispatch/--recursive remain compatibility options; traversal and dispatch are already enabled. --sync remains an explicit diagnostic execution scope.

An application can opt in individual files during StorageNodeDiscoveredEvent by setting $event->node->meta[StorageNode::SCAN_REQUEST][StorageNode::LOAD_FILES] = true. Use a listener priority above zero so this happens before NodeRefreshListener decides whether an existing file needs loading. The node then records its effective mode; its siblings keep the root's mode.

Directly persisted new file nodes and tracked-write journal processing still use load_file; listing-only is a scan policy, not a prohibition on explicit file loads. Already queued file jobs are not cancelled by changing the scan default. Inspect and drain or selectively remove old scan jobs before a controlled listing-only run; do not purge unrelated messages. Reports must recognize file_listed as complete for listing-only work. The new request fields use the existing JSON meta column; they need no new inventory table or schema migration. The separate BIGINT file-size mapping requires consuming apps to migrate storage_node.file_size if still INTEGER.

Applications may listen for StorageNodeDiscoveredEvent during listing or StorageFileLoadedEvent after loading, before the executor flushes. No AI-specific listener or association to an original object is needed to cache a sidecar. Mediary keeps one file per media ID and task, at the existing MediaKeyService::archivePathFromKey($id, $task.'.json', 'o') path. Its CachedSidecarReader derives that path and decodes the cached node contents. The S3 payload is unchanged. Tasks remain independent because they are different files. A cache miss falls back to storage; a cache hit performs no S3 request.

Tracked writes and changelog

track: true wraps the Flysystem adapter so directly injected operators and StorageService writes are tracked alike. Successful writes, streams, moves, copies, deletes and visibility changes append durable Doctrine journal rows. Failed backend writes and reads append nothing. Apply the application migrations for StorageChange and StorageJournalHead before enabling tracking. Journal rows have no foreign keys to disposable StorageNodes. Redis is not required.

storage:changes:consume forwards journal entries to the same StorageNode workflow and marks them dispatched. It never loads files itself. Mediary's Scheduler runs this forwarding command every minute. A crash after dispatch can produce a safe duplicate invalidation. Workers read the current file, so delayed messages do not restore old content. Missing files are marked meta[StorageNode::EXISTS] = false and their cached contents are removed. Per-node workflow locks serialize updates.

The admin-only GET /storage/changes?after=0&limit=100 endpoint returns events and a next cursor. Without authorization it returns 403. Sequence allocation is serialized through database commit; rows remain available after dispatch and clients checkpoint after processing a page. There is no automatic pruning.

Filesystem writes and SQL journal inserts cannot be atomic. Journal failures are reported even if the file was saved: retry or explicitly record the successful write. Writes bypassing Flysystem require provider notifications or an explicit receipt; timestamps can detect changes on subsequent listings, but cannot prove content identity. ETags are opaque, not necessarily content hashes.

$journal->record('archive.storage', $relativeKey, revision: [
    SidecarCache::ETAG => $etag,
    SidecarCache::VERSION_ID => $versionId,
]);
// Maps confirmed bucket/key receipts through configured zone prefixes:
$externalStorageWrite->record($bucket, $key, $revision);

storage:changes:record archive.storage path/to/object checks existence before recording a manual receipt; --etag and --version-id are optional. Notification provenance is saved under meta.storage_event_revision, separately from the hash of the actual cached contents. Existing AI response provenance is left unchanged.

Mediary records Imagor originals only after its existing HEAD confirms persistence, including ETag/version when available. Resize HTTP success alone does not identify or confirm a persisted result object; its producer must supply a confirmed key before using the same recording API. No guessed resize keys are journaled.

Browser visibility

survos_storage.storages configures tracking (track, sidecar_prefix, content_patterns); Flysystem still defines the storage connections. The optional survos_storage.browse_storages list selects service IDs for the storage overview and navbar. Omit it or use [] to show all configured stores. It is a presentation filter, not access control: underlying services and direct storage routes remain available.

survos_storage:
    browse_storages: [local_vault.storage, platform_dfolio.storage]

Friendly storage names

Flysystem service IDs and bucket/prefix configuration remain in flysystem.yaml. Put app-specific display text in translations/storage.en.yaml:

label:
    'platform_folio.storage': 'Published folios · S3'
zone:
    'platform_folio.storage': 'DuckDB .dfolio snapshots and legacy database artifacts.'

label.<storage-id> appears in the navbar, storage switcher and listings, falling back to the storage ID when untranslated. Existing zone.<storage-id> keys are descriptions, displayed on both the overview and directory page. Directory views provide breadcrumbs, folder-first ordering and retain signed-link mode when navigating. These labels do not rename services, buckets, prefixes or objects.

survos_storage.yaml controls tracking (track), sidecar locations (sidecar_prefix), scanned content (content_patterns) and the tracking lock factory. It does not define storage connections or presentation labels.