Keeping Knowledge Alive
A temporal knowledge graph can help an agent find the right source, but first someone has to build and maintain the map. That work is larger than exporting Workspace documents. Enterprise knowledge is spread across databases, warehouses, document systems, APIs, code, and applications such as Salesforce, and each system exposes different identities, metadata, event models, permissions, and evidence about what changed.
The initial instinct is to treat this as a migration: point a large number of agents at every source, wait for them to finish, and publish one complete Open Knowledge Format bundle. The premise is attractive and the boundary is wrong. In a sufficiently large organization, the earliest sources will change before the final sources have been processed. The first build is already a maintenance problem.
This paper therefore treats ingestion and regeneration as two phases of one knowledge-production system. Part I defines how a bounded fleet of producers builds useful, reviewable OKF generations progressively. Part II defines how each source triggers later regeneration through events, scheduled comparison, or both. In both phases, source content stays in its system of record; the knowledge layer holds portable context, provenance, temporal claims, and pointers back to the authoritative object.
The work is complete as an architecture and operating study, informed by evaluations I have run against enterprise data. It establishes the production model, component boundaries, failure controls, and validation criteria. What those evaluations showed is reported as observation, not as benchmark results, so no performance percentages are asserted.
A map that begins behind reality
Initial ingestion faces three forms of drift before its first stable release. Source drift occurs when a table, document, record type, permission, or API changes while the corpus is being scanned. Semantic drift occurs when a business definition or ownership relationship changes after its technical object was observed. Coverage drift occurs when the organization adds a new source or discovers that a connector never enumerated part of its scope.
A one-shot crawler hides all three. It produces files, but it cannot say which parts of the enterprise were observed, when, or whether an early concept still matches its source at publication time. A usable bootstrap therefore needs the same properties as steady-state maintenance:
- stable source identities and revisions;
- explicit observation and generation times;
- repeatable, idempotent producers;
- a coverage manifest that shows what was and was not scanned;
- immutable candidate and published generations; and
- a path for re-observing objects that changed during the run.
The OKF v0.2 specification
already provides a portable concept structure, source pointers, generation and
verification signals, lifecycle state, and stale_after. It deliberately does
not prescribe how a source is inventoried, how an LLM is orchestrated, or how
a bundle is kept current. Those decisions belong to the producer system
described here.
Research frame
One lifecycle, two operating phases
Inventory, draft, validate, and publish useful slices instead of waiting for one terminal crawl.
Use a source-specific trigger and reconciliation policy after each slice becomes live.
The originating system remains authoritative for content and final access control.
The two phases share source adapters, concept keys, validators, and the publication contract, so the bootstrap is not disposable code and the maintenance system does not rediscover every decision it made.
Part I · Build progressively, not all at once
The initial objective is the smallest useful and defensible description of each approved source object, enriched only where more context improves retrieval.
Begin with a source inventory
Each connector first enumerates objects and emits observations under the source’s existing authorization boundary. The observation surface varies by system:
| Source class | Useful initial evidence | Content that should remain at source |
|---|---|---|
| Databases and warehouses | catalogs, schemas, columns, constraints, labels, descriptions, stable object IDs | business rows and protected payloads |
| Content systems | file IDs, titles where permitted, MIME type, revision, owner signals, approved excerpts | complete documents unless explicitly authorized for processing |
| Enterprise applications | object schemas, field definitions, record types, API resources, lifecycle metadata | customer and operational records |
| Code and API systems | repository paths, symbols, API schemas, releases, ownership files | secrets, private runtime state, and unrelated source content |
Google Cloud’s Knowledge Catalog automatically collects technical metadata from several Google data systems and supports third-party sources, which reduces the inventory work for sources already visible there. Business applications, content repositories, custom APIs, and organization-specific semantics still need their own authorized adapters.
The OKF reference agent shows a useful narrow pattern: it writes concepts from BigQuery metadata, then runs a separate LLM-assisted pass over approved web sources to enrich them. This paper generalizes that pattern into a fleet of replaceable producers without treating the proof of concept as a finished enterprise ingestion service.
Publish in slices
The coordinator partitions the inventory by source, security domain, object type, and priority. A first generation might cover certified warehouse tables. A second might add their definitions and approved documentation. Later generations can add Workspace policies, Salesforce object models, operational runbooks, and cross-system relationships.
Figure 1 · Progressive initial ingestion
Progressive publication makes the layer useful sooner, but only if partial coverage is visible. A producer-defined coverage manifest records the sources and object classes requested, observed, excluded, failed, awaiting review, and published. It is an implementation extension, not a standard OKF field, and it must never claim that an unobserved system contains no relevant knowledge.
Separate observation from interpretation
Each source object moves through explicit stages:
- Discover. Record the stable source identity and the security domain in which it was observed.
- Observe. Read the current authoritative metadata and record its revision, modification signal, and a normalized digest.
- Draft. Map deterministic facts into an OKF candidate.
- Enrich. Ask a model for descriptions, relationships, or usage context only when approved evidence is available.
- Validate. Check format, links, identity, provenance, temporal conflicts, and publication policy.
- Review or publish. Automatically publish safe source-derived facts; route semantic assertions and sensitive changes to an owner.
This keeps a model from becoming the source of truth for facts already available through an API, and it leaves a durable record of which claims were observed, which were synthesized, and which were confirmed.
A bounded producer fleet, not an ungoverned swarm
Large initial inventories benefit from parallel work, but the scalable unit is a partitioned task, not an unconstrained autonomous agent. Thousands of agents independently exploring the same systems would duplicate reads, compete for API quotas, produce inconsistent concepts, and make publication order nondeterministic. The fleet instead has bounded roles:
| Role | Primary behavior | Model requirement |
|---|---|---|
| Inventory coordinator | partitions approved scope and records coverage | deterministic orchestration |
| Source adapter | enumerates and rereads authoritative metadata | deterministic API client |
| Concept producer | maps observations into structured OKF candidates | templates first; model when semantics are needed |
| Relationship investigator | evaluates a small evidence-backed candidate set | Flash-class model with tools |
| Validator | checks schema, links, provenance, temporal rules, and policy | deterministic code |
| Review router | assigns ambiguous or governed changes to an owner | rules, with model-assisted prioritization |
Google’s Agent Development Kit supports multi-agent orchestration, and Cloud Run jobs can execute many independent tasks in parallel. The architecture uses each where it fits: ADK when a worker must reason, call tools, and gather evidence; Cloud Run jobs, Workflows, Pub/Sub, and Cloud Tasks for durable partitioning, bounded concurrency, retry, and source rate limits.
Use a model ladder
A fast model does not make every knowledge decision, and a frontier reasoning model does not copy a table schema. The current Google model catalog describes Gemini 3.5 Flash-Lite as optimized for high-throughput classification, routing, JSON extraction, document understanding, and lightweight agentic work, which makes it a credible candidate for routine concept drafting. The same documentation recommends higher thinking levels when it runs as an autonomous subagent with tools.
Gemini 3.7 Flash, which reached general availability in August 2026, is positioned as the primary agentic workhorse in the Gemini 3 family. It is the escalation tier for ambiguous relationships, conflicting sources, and multi-document synthesis. A human remains the final escalation for governed meaning, authority, or temporal validity that the evidence does not establish.
Both tiers support structured output. A response schema makes the candidate machine-readable; it does not prove the values are true, so deterministic validation and evidence links remain mandatory. For the non-interactive initial backlog, Gemini batch inference can process Cloud Storage or BigQuery inputs asynchronously. Targeted online calls fit better after an event marks one object dirty.
Give each concept one publication owner
Parallel workers need a deterministic identity and a single-writer rule. A practical idempotency key combines the source system, stable object ID, observed revision or digest, and producer version. Reprocessing the same key produces the same candidate or no change. Workers explore in parallel, but only the publication coordinator advances the current generation.
Candidate outputs are immutable. If the source changes while a worker is running, the worker rereads the source revision before publication and abandons the stale candidate. If two workers propose different semantic interpretations, the system keeps both drafts and routes the conflict rather than letting the last write win.
Part II · Regenerate by source policy
Once a concept is published, maintenance begins with a source signal: an event, a change-data-capture record, a webhook, an audit entry, or the result of a scheduled inventory comparison. The signal is a reason to reread the source, not automatically the new truth. Different systems expose different signals:
- Google Workspace can deliver Drive events through Pub/Sub, while the Drive Changes API provides page tokens for reconciliation.
- Databases can expose metadata snapshots, audit records, DDL signals, or change-data-capture streams. Google Cloud Datastream currently supports multiple relational databases and several enterprise application sources where replicating change data is already an approved requirement.
- Salesforce exposes object metadata through Describe and ObjectInfo APIs. Its Change Data Capture events and Pub/Sub replay window are useful when a concept actually depends on record-level change.
- Enterprise systems without a reliable event surface require scheduled API inventory, normalized digests, and bounded comparison.
Change data capture does not mean copying every row into the knowledge pipeline. A table-row update should not regenerate a schema concept unless that concept carries a data-freshness, volume, or operational claim the update affects. Datastream can supply an existing CDC path, but deploying replication solely to refresh OKF would usually violate the metadata-first design.
The connector contract hides these differences from the OKF producer. It emits a source observation containing identity, revision, observation time, normalized facts, access partition, and evidence, and the producer applies the same candidate, validation, and publication path used during bootstrap.
Let operators choose two independent controls
“Event-only” and “periodic maintenance” are not mutually exclusive settings. The useful configuration has two knobs: what triggers targeted regeneration, and how often a broader reconciliation sweep verifies coverage.
| Regeneration trigger | Reconciliation cadence | Appropriate use |
|---|---|---|
| Event | Off | Low-risk sources with a complete enough event contract and an accepted gap risk |
| Event | Daily or weekly | Fast targeted updates with a repair path; the recommended default for important sources |
| Scheduled snapshot | Same schedule | Sources without events or where batch comparison is simpler and safer |
| Manual approval window | Scheduled detection | Policies, definitions, and other governed semantics that must not self-publish |
An event-triggered system with periodic cleanup is therefore a hybrid policy: events provide low-latency regeneration, and the sweep repairs missed events, expired subscriptions, incomplete source coverage, and changes that do not emit the expected signal. Operators can choose pure event-driven operation, but the system makes the residual risk explicit and sets conservative staleness behavior.
The policy belongs to each source class, not the whole corpus. A rapidly changing table schema might use events plus nightly reconciliation. A stable business glossary might use a weekly snapshot and mandatory review. A Salesforce object could use scheduled Describe or ObjectInfo reads for schema, Change Data Capture for approved record-dependent freshness claims, and a separate review schedule for semantic guidance.
Regenerate only what became dirty
After a signal arrives, the maintenance path is narrow:
- coalesce signals for the same source identity;
- reread the object with a source-scoped identity;
- compare the authoritative revision and normalized digest with the ledger;
- stop when nothing meaningful changed;
- regenerate only the affected observed concept and mark linked semantic concepts dirty;
- apply deterministic validation and the review policy; and
- publish a complete new generation, then update its search and graph projections.
Source-derived facts can usually move automatically: canonical identifiers, URIs, schemas, field types, revision timestamps, and reproducible usage statistics. Business definitions, source authority, ownership, validity intervals, supersession, classification, and human verification require explicit evidence or review. A regenerated concept must not silently keep an earlier human-reviewed label unless that verification was bound to unchanged content.
One Google Cloud implementation
The following architecture is one way to implement the model on Google Cloud. The OKF bundle remains the portable boundary, so the source connectors, orchestration services, models, state store, review interface, and consumer can each be replaced independently.
Figure 2 · Progressive ingestion and continuous regeneration on Google Cloud
Authoritative enterprise systems feed initial inventory, continuous event signals, and scheduled reconciliation. These converge on a bounded Google Cloud producer fleet, then pass through evidence capture, deterministic validation, review, and atomic Open Knowledge Format publication before governed retrieval and measurement.
01 · Authoritative systems
02 · Inventory + maintain
03 · Produce
04 · Review + publish
05 · Serve + measure
Map the services to responsibilities
| Responsibility | Google Cloud surface | Reason for the boundary |
|---|---|---|
| Coordinate bounded phases | Workflows | explicit ordering, parallel branches, retries, and auditable execution state |
| Run partitioned producers | Cloud Run jobs and services | independent tasks for bootstrap; request-driven workers for incremental refresh |
| Fan out and rate-limit work | Pub/Sub and Cloud Tasks | durable signals, retry, dead letters, and source-specific concurrency control |
| Schedule repair | Cloud Scheduler | subscription renewal, inventory comparison, and stale-concept scans |
| Draft structured candidates | Gemini Flash-Lite and Flash on Agent Platform | model ladder for routine extraction and harder evidence synthesis |
| Hold operational state | transactional ledger plus BigQuery history | current cursors and leases separated from append-only experiment evidence |
| Publish portable knowledge | version control or Cloud Storage generations | immutable candidates, reviewable diffs, and rollback |
| Govern and retrieve | Knowledge Catalog and optional custom projections | serve approved context without coupling it to the producer runtime |
Cloud Run array jobs execute independent tasks in parallel, with parallelism kept below the quotas of the source and model APIs they call. Workflows coordinates concurrent branches, and Cloud Tasks provides per-queue dispatch and concurrency limits. Pub/Sub’s ordinary delivery is at-least-once, so every consumer still needs the idempotency and source-revision checks above.
Security partitions survive every stage. A broadly readable OKF bundle cannot safely contain metadata harvested under a more privileged identity. Source adapters collect the minimum useful metadata, workers use separate least-privileged identities, and consumers reauthorize against the source before retrieving content. Trust fields in OKF describe evidence; they do not grant access.
Design conclusions and validation framework
The study compared two connected operating problems: building the first useful corpus and keeping it current. The conclusions are design decisions with stated rationale, grounded in documented source behavior, service contracts, known failure modes, and what I observed while evaluating the design against enterprise data. The measures beside them are the acceptance framework for a deployment, not measured results.
Initial-ingestion conclusions
Three ingestion patterns were weighed against the same approved multi-system scope: a serial human-assisted producer; one general-purpose agent processing sources sequentially; and the bounded producer fleet with deterministic adapters, Flash-Lite drafting, Flash escalation, validation, and review routing. The fleet was selected because it preserves deterministic source reads, single-writer publication, and explicit review while still partitioning the inventory for parallel execution. The serial patterns remain useful baselines, but neither combines progressive coverage with source-specific concurrency control.
The validation corpus combines database schemas, content-system documents, and an enterprise-application sandbox with deliberately overlapping concepts, and its workload introduces changes during ingestion to verify that progressive generations converge instead of freezing an inconsistent snapshot. Acceptance is measured by time to first useful coverage, time to declared coverage, valid concepts per thousand source objects, source and relationship precision, human review minutes, model cost per accepted concept, duplicate or conflicting concept rate, permission-leakage count, and the value available after each partial generation.
Regeneration conclusions
Three maintenance policies were compared architecturally: periodic or manual refresh; event-driven regeneration without broad reconciliation; and event-driven regeneration with scheduled reconciliation and review gates. The hybrid policy was selected as the default for important sources. Events give the shortest path to a targeted refresh, and reconciliation repairs documented gaps such as expired subscriptions, limited replay windows, duplicate delivery, and incomplete change coverage. Event-only operation remains an explicit lower-cost choice for sources whose owners accept its gap risk; scheduled snapshots remain the correct path where reliable events do not exist.
The validation workload introduces schema changes, document revisions, enterprise-object changes, permission revocations, duplicate and out-of-order messages, expired subscriptions, temporary connector failure, and ambiguous supersession. It records source-to-publication lag, change-capture recall, stale-concept exposure, incorrect automatic publication, reconciliation repair yield, human-review load, revocation-to-index-removal time, and recovery after an outage.
Four conclusions follow:
- C1: the bounded producer fleet is the correct unit of parallelism because it increases concurrency without surrendering deterministic identity, validation, or single-writer publication;
- C2: deterministic extraction paired with a Flash model ladder aligns model capability with task complexity and avoids spending reasoning capacity on facts already available through APIs;
- C3: event-driven regeneration improves timeliness, but reconciliation is the mechanism that restores confidence after event-path gaps; and
- C4: progressive generations are safe to consume before full coverage only when their coverage manifest makes unobserved and failed scope explicit.
The measures also say when the design is failing. Review load growing linearly with every source change indicates a weak semantic boundary. Reconciliation finding almost no drift supports a lower cadence. Event-only operation missing important changes disqualifies it as an equivalent to the hybrid policy. A parallel fleet generating more conflicts than the serial baseline shows that work was partitioned at the wrong boundary.
Scope and evidence boundaries
The reviewed Google product documentation does not describe this maintenance plane as one managed feature. The OKF reference agent is a proof of concept for BigQuery and approved web enrichment; the multi-source producer fleet is the contribution developed here. Knowledge Catalog can collect and enrich metadata, but an implementation still has to define OKF generation, review, temporal semantics, and source-specific maintenance policy.
Modification time is not business-valid time. A model can detect that two documents look related, but similarity does not prove that one supersedes the other. An event can reveal that an object changed, but it does not prove that the event stream is complete. Structured output can guarantee shape, not truth. These boundaries are the reason the design keeps deterministic observation, model interpretation, and governed publication separate.
The result is not a knowledge layer that claims to be finished. It is a layer that states what it has observed, publishes useful increments, detects when its evidence changes, and regenerates at a cost and risk level chosen for each source. Initial ingestion creates the first trustworthy generation. Maintenance keeps that generation from becoming a confidently stale map.