Rolling main documentation

Built from commit f80b604a992ba1144e6c45229a8165c81e52dae0. Content may describe unreleased behavior.

Edge Cookie External Sync Architecture

Trusted Server supports ID syncing. Publishers who have engaged in an agreement with identity partners can use our sync endpoints found below.

Overview

Architecture Principles

Component	Role	Characteristics
KV Store	Hot cache	Fast reads (~1ms), edge-local, eventually consistent
Object Store	Source of truth	Durable, supports range queries, sync endpoint
Secret Key	Shared salt	Distributed out-of-band, enables a data sharing agreement

Configuration

Add the following to your trusted-server.toml:

[collective]
enabled = true
sync_endpoint = "https://collective.example.com/sync"

# KV Store names (must match fastly.toml)
kv_store = "collective_store"

Fastly.toml Setup

[local_server.kv_stores]
    [[local_server.kv_stores.collective_store]]
        key = "placeholder"
        data = "placeholder"

Data Model

KV Store Record

Each EC ID maps to a compact JSON record optimized for fast reads:

{
  "sid": "0f99d7dc...a98e.45np22",
  "seg": ["auto-intender", "sports-fan"],
  "lst": 1706470800,
  "src": ["pub-a.com", "pub-b.com"]
}

Field	Type	Description
sid	string	Full EC ID (64hex.6alnum)
seg	string[]	Audience segments
lst	integer	Last seen timestamp (Unix epoch)
src	string[]	Contributing publisher domains

Object Store Record (Source of Truth)

The Object Store maintains a richer record with full history:

{
  "ec_id": "0f99d7dc...a98e.45np22",
  "hmac_base": "0f99d7dc...a98e",
  "random_suffix": "45np22",
  "segments": ["auto-intender", "sports-fan"],
  "last_seen": "2024-01-28T15:00:00Z",
  "first_seen": "2024-01-15T10:30:00Z",
  "sources": [
    { "domain": "pub-a.com", "last_seen": "2024-01-28T15:00:00Z" },
    { "domain": "pub-b.com", "last_seen": "2024-01-27T12:00:00Z" }
  ],
  "version": 3
}

Sync Protocol

Initial Sync

When a new publisher has a new agreement, they perform a full sync:

GET /sync?type=full
Authorization: Bearer <collective-token>

Response (NDJSON stream for large datasets):

{"ec_id": "abc123.x1y2z3", "segments": [...], "last_seen": "..."}
{"ec_id": "def456.a1b2c3", "segments": [...], "last_seen": "..."}

Incremental Updates

Subsequent syncs only fetch changes since last sync:

GET /sync?type=incremental&since=1706470800
Authorization: Bearer <collective-token>

Response includes only records modified after the since timestamp:

{
  "records": [
    {"ec_id": "abc123.x1y2z3", "segments": [...], "last_seen": "..."}
  ],
  "next_cursor": "1706475000",
  "has_more": false
}

Write Operations

When a publisher observes new data, they write to the Object Store:

POST /sync
Authorization: Bearer <collective-token>
Content-Type: application/json

{
  "records": [
    {
      "ec_id": "abc123.x1y2z3",
      "segments": ["new-segment"],
      "source_domain": "pub-a.com"
    }
  ]
}

The sync endpoint handles:

• Deduplication by ec_id
• Segment merging (union of all observed segments)
• Source tracking (which publishers contributed data)
• Version increment for conflict resolution

Request Flow

Read Path (Hot)

1. Browser request arrives at edge
2. Extract/generate EC ID
3. KV Store lookup by EC ID
4. If hit: return cached segments
5. If miss: fetch from Object Store, populate KV, return

Write Path (Async)

1. Observe new user data (segment, behavior)
2. Queue write to Object Store
3. Object Store updates source of truth
4. Partner instances poll for updates
5. Partners update their KV caches

Privacy Considerations

• Identifier content: EC IDs are one-way HMAC values with a random suffix and embed no readable personal fields
• Consent-gated: Inclusion follows the consent signals and policy the publisher configures
• Publisher control: Each publisher controls what segments they share
• Audit trail: Object Store maintains full history of data sources

Implementation Status

Feature	Status
KV Store integration	✅ Available
Object Store writes	🚧 In development
Sync endpoint	🚧 In development
Incremental sync	📋 Planned
Segment merging	📋 Planned

Next Steps

• Edge Cookies - Understand ID generation
• Configuration Reference - Full config options
• GDPR Compliance - Consent signal handling