Keeping Knowledge Alive

A temporal knowledge graph can help an agent find the right source, but first someone has to build and maintain the map. That work is larger than exporting Workspace documents. Enterprise knowledge is distributed across databases, warehouses, document systems, APIs, code, and applications such as Salesforce. Each system exposes different identities, metadata, event models, permissions, and evidence about what changed.

The initial instinct is to treat this as a migration: point a large number of agents at every source, wait for them to finish, and publish one complete Open Knowledge Format bundle. The premise is attractive and the boundary is wrong. In a sufficiently large organization, the earliest sources will change before the final sources have been processed. The first build is already a maintenance problem.

This paper therefore establishes ingestion and regeneration as two phases of one knowledge-production system. Part I defines how a bounded fleet of producers builds useful, reviewable OKF generations progressively. Part II defines how each source triggers later regeneration through events, scheduled comparison, or a hybrid of the two. In both phases, source content remains in its system of record. The knowledge layer stores portable context, provenance, temporal claims, and pointers back to the authoritative object.

The work documented here is complete as an architecture and operating study. It establishes the production model, component boundaries, failure controls, and validation criteria. Quantitative results from a deployed benchmark fall outside the evidence collected for this study, so no unmeasured performance percentages are asserted.

A map that begins behind reality

Initial ingestion has three forms of drift before it reaches its first stable release.

Source drift occurs when a table, document, record type, permission, or API changes while the corpus is being scanned. Semantic drift occurs when a business definition or ownership relationship changes after its technical object was observed. Coverage drift occurs when the organization adds a new source or discovers that an existing connector never enumerated part of its scope.

A one-shot crawler hides all three. It produces files, but it cannot say which parts of the enterprise were observed, when they were observed, or whether an early concept still matches its source at publication time. A usable bootstrap therefore needs the same properties as steady-state maintenance:

• stable source identities and revisions;
• explicit observation and generation times;
• repeatable, idempotent producers;
• a coverage manifest that shows what was and was not scanned;
• immutable candidate and published generations; and
• a path for re-observing objects that changed during the run.

The OKF v0.2 specification already provides a portable concept structure, source pointers, generation and verification signals, lifecycle state, and stale_after. It intentionally does not prescribe how a source is inventoried, how an LLM is orchestrated, or how a bundle is kept current. Those operating decisions belong to the producer system described here.

Research frame

One lifecycle, two operating phases

01 Progressive bootstrap

Inventory, draft, validate, and publish useful slices instead of waiting for one terminal crawl.

02 Continuous regeneration

Use a source-specific trigger and reconciliation policy after each slice becomes live.

1 Authority boundary

The originating system remains authoritative for content and final access control.

The two phases use the same source adapters, concept keys, validators, and publication contract. This prevents the bootstrap from becoming disposable code and the maintenance system from rediscovering every decision it made.

Part I · Build progressively, not all at once

The initial objective is not “read everything.” It is to produce the smallest useful and defensible description of each approved source object, then enrich that description only where more context improves retrieval.

Begin with a source inventory

Each connector first enumerates objects and emits observations under the source’s existing authorization boundary. The observation surface varies by system:

Source class	Useful initial evidence	Content that should remain at source
Databases and warehouses	catalogs, schemas, columns, constraints, labels, descriptions, stable object IDs	business rows and protected payloads
Content systems	file IDs, titles where permitted, MIME type, revision, owner signals, approved excerpts	complete documents unless explicitly authorized for processing
Enterprise applications	object schemas, field definitions, record types, API resources, lifecycle metadata	customer and operational records
Code and API systems	repository paths, symbols, API schemas, releases, ownership files	secrets, private runtime state, and unrelated source content

Google Cloud’s current Knowledge Catalog automatically collects technical metadata from several Google data systems and supports third-party sources. That can reduce the inventory work for sources already visible there. It does not eliminate the need for source-specific producers: business applications, content repositories, custom APIs, and organization-specific semantics still require their own authorized adapters.

The OKF reference agent demonstrates a useful narrow pattern. It first writes concepts from BigQuery metadata, then performs a separate LLM-assisted pass over approved web sources to enrich those concepts. This paper generalizes that pattern into a fleet of replaceable producers without treating the proof of concept as a finished enterprise ingestion service.

Publish in slices

The coordinator partitions the inventory by source, security domain, object type, and priority. A first generation might cover certified warehouse tables. A second might add their definitions and approved documentation. Later generations can add Workspace policies, Salesforce object models, operational runbooks, and cross-system relationships.

Figure 1 · Progressive initial ingestion

Observe Inventory approved source objects

→

Draft Generate bounded candidate concepts

→

Release Validate and publish a declared slice

Progressive publication makes the layer useful sooner, but only if partial coverage is visible. A producer-defined coverage manifest should record the sources and object classes requested, observed, excluded, failed, awaiting review, and published. This manifest is an implementation extension, not a standard OKF field. It must never claim that an unobserved system contains no relevant knowledge.

Separate observation from interpretation

Each source object should move through explicit stages:

1. Discover. Record the stable source identity and the security domain in which it was observed.
2. Observe. Read the current authoritative metadata and record its revision, modification signal, and a normalized digest.
3. Draft. Map deterministic facts into an OKF candidate.
4. Enrich. Ask a model for descriptions, relationships, or usage context only when approved evidence is available.
5. Validate. Check format, links, identity, provenance, temporal conflicts, and publication policy.
6. Review or publish. Automatically publish safe source-derived facts; route semantic assertions and sensitive changes to an owner.

This separation prevents a model from becoming the source of truth for facts already available through an API. It also creates a durable record of which claims were observed, which were synthesized, and which were confirmed.

A bounded producer fleet, not an ungoverned swarm

Large initial inventories benefit from parallel work, but the scalable unit is usually a partitioned task—not an unconstrained autonomous agent. Thousands of agents independently exploring the same systems would duplicate reads, compete for API quotas, produce inconsistent concepts, and make publication order nondeterministic.

The resulting fleet has bounded roles:

Role	Primary behavior	Model requirement
Inventory coordinator	partitions approved scope and records coverage	deterministic orchestration
Source adapter	enumerates and rereads authoritative metadata	deterministic API client
Concept producer	maps observations into structured OKF candidates	templates first; model when semantics are needed
Relationship investigator	evaluates a small evidence-backed candidate set	Flash-class model with tools
Validator	checks schema, links, provenance, temporal rules, and policy	deterministic code
Review router	assigns ambiguous or governed changes to an owner	rules, with model-assisted prioritization

Google’s Agent Development Kit supports multi-agent orchestration, while Cloud Run jobs can execute many independent tasks in parallel. The architecture uses those capabilities selectively: ADK is useful when a worker must reason, call tools, and gather evidence; Cloud Run jobs, Workflows, Pub/Sub, and Cloud Tasks are better suited to durable partitioning, bounded concurrency, retry, and source rate limits.

Use a model ladder

A fast model does not make every knowledge decision, and a frontier reasoning model does not copy a table schema. The current Google model catalog describes Gemini 3.5 Flash-Lite as optimized for high-throughput classification, routing, JSON extraction, document understanding, and lightweight agentic work. That makes it a credible candidate for routine concept drafting. The same documentation recommends higher thinking levels when it is used as an autonomous subagent with tools.

Gemini 3.7 Flash, which reached general availability in August 2026, is positioned as the primary agentic workhorse in the Gemini 3 family. It can be the escalation tier for ambiguous relationships, conflicting sources, and multi-document synthesis. A human remains the final escalation for governed meaning, authority, or temporal validity that the evidence does not establish.

Both tiers support structured output. A response schema makes the candidate machine-readable, but it does not prove that the values are true. Deterministic validation and evidence links remain mandatory. For the non-interactive initial backlog, Gemini batch inference can process Cloud Storage or BigQuery inputs asynchronously; targeted online calls are more appropriate after an event marks one object dirty.

Give each concept one publication owner

Parallel workers need a deterministic identity and a single-writer rule. A practical idempotency key combines the source system, stable object ID, observed revision or digest, and producer version. Reprocessing the same key produces the same candidate or no change. Workers can explore in parallel, but only the publication coordinator advances the current generation.

Candidate outputs are immutable. If the source changes while a worker is running, the worker rereads the source revision before publication and abandons the stale candidate. If two workers propose different semantic interpretations, the system preserves both drafts and routes the conflict rather than allowing the last write to win.

Part II · Regenerate by source policy

Once a concept is published, maintenance begins with a source signal. That signal can be an event, a change-data-capture record, a webhook, an audit entry, or the result of a scheduled inventory comparison. It is treated as a reason to reread the source—not automatically as the new truth.

Different systems expose different signals:

• Google Workspace can deliver Drive events through Pub/Sub, while the Drive Changes API provides page tokens for reconciliation.
• Databases can expose metadata snapshots, audit records, DDL signals, or change-data-capture streams. Google Cloud Datastream currently supports multiple relational databases and several enterprise application sources where replicating change data is already an approved requirement.
• Salesforce exposes object metadata through Describe and ObjectInfo APIs. Its Change Data Capture events and Pub/Sub replay window are useful when a concept actually depends on record-level change.
• Enterprise systems without a reliable event surface require scheduled API inventory, normalized digests, and bounded comparison.

Change data capture is not a requirement to copy every row or business record into the knowledge pipeline. A table-row update should not regenerate a schema concept unless that concept contains a data-freshness, volume, or operational claim affected by the update. Datastream can supply an existing CDC path, but deploying replication solely to refresh OKF would usually violate the narrower metadata-first design.

The connector contract hides these differences from the OKF producer. It emits a source observation containing identity, revision, observation time, normalized facts, access partition, and evidence. The producer then applies the same candidate, validation, and publication path used during bootstrap.

Let operators choose two independent controls

“Event-only” and “periodic maintenance” are not mutually exclusive settings. The useful configuration has two knobs: what triggers targeted regeneration, and how often a broader reconciliation sweep verifies coverage.

Regeneration trigger	Reconciliation cadence	Appropriate use
Event	Off	Low-risk sources with a complete enough event contract and an accepted gap risk
Event	Daily or weekly	Fast targeted updates with a repair path; the recommended default for important sources
Scheduled snapshot	Same schedule	Sources without events or where batch comparison is simpler and safer
Manual approval window	Scheduled detection	Policies, definitions, and other governed semantics that must not self-publish

An event-triggered system with periodic cleanup is therefore a hybrid policy: events provide low-latency regeneration and the sweep repairs missed events, expired subscriptions, incomplete source coverage, and changes that do not emit the expected signal. Operators can choose pure event-driven operation, but the system makes its residual risk explicit and sets conservative staleness behavior.

The policy belongs to each source class rather than the whole corpus. A rapidly changing table schema might use events plus nightly reconciliation. A stable business glossary might use a weekly snapshot and mandatory review. A Salesforce object could use scheduled Describe or ObjectInfo reads for schema, Change Data Capture for approved record-dependent freshness claims, and a separate review schedule for semantic guidance.

Regenerate only what became dirty

After receiving a signal, the maintenance path is narrow:

1. coalesce signals for the same source identity;
2. reread the object with a source-scoped identity;
3. compare the authoritative revision and normalized digest with the ledger;
4. stop when nothing meaningful changed;
5. regenerate only the affected observed concept and mark linked semantic concepts dirty;
6. apply deterministic validation and the review policy; and
7. publish a complete new generation, then update its search and graph projections.

Source-derived facts can usually move automatically: canonical identifiers, URIs, schemas, field types, revision timestamps, and reproducible usage statistics. Business definitions, source authority, ownership, validity intervals, supersession, classification, and human verification require explicit evidence or review. A changed generated concept must not silently retain an earlier human-reviewed label unless that verification was bound to unchanged content.

One Google Cloud implementation

The following architecture is one way to implement the research model on Google Cloud. The OKF bundle remains the portable boundary: the source connectors, orchestration services, models, state store, review interface, and consumer can be replaced independently.

Figure 2 · Progressive ingestion and continuous regeneration on Google Cloud

Scroll horizontally to inspect the architecture →

Map the services to responsibilities

Responsibility	Google Cloud surface	Reason for the boundary
Coordinate bounded phases	Workflows	explicit ordering, parallel branches, retries, and auditable execution state
Run partitioned producers	Cloud Run jobs and services	independent tasks for bootstrap; request-driven workers for incremental refresh
Fan out and rate-limit work	Pub/Sub and Cloud Tasks	durable signals, retry, dead letters, and source-specific concurrency control
Schedule repair	Cloud Scheduler	subscription renewal, inventory comparison, and stale-concept scans
Draft structured candidates	Gemini Flash-Lite and Flash on Agent Platform	model ladder for routine extraction and harder evidence synthesis
Hold operational state	transactional ledger plus BigQuery history	current cursors and leases separated from append-only experiment evidence
Publish portable knowledge	version control or Cloud Storage generations	immutable candidates, reviewable diffs, and rollback
Govern and retrieve	Knowledge Catalog and optional custom projections	serve approved context without coupling it to the producer runtime

Cloud Run array jobs execute independent tasks in parallel, with their parallelism kept below the quotas of the source and model APIs they call. Workflows coordinate concurrent branches, while Cloud Tasks provides per-queue dispatch and concurrency limits. Pub/Sub’s ordinary delivery is at-least-once, so every consumer still needs the idempotency and source-revision checks described above.

Security partitions survive every stage. A broadly readable OKF bundle cannot safely contain metadata harvested under a more privileged identity. Source adapters collect the minimum useful metadata, workers use separate least-privileged identities, and consumers reauthorize against the source before retrieving content. Trust fields in OKF describe evidence; they do not grant access.

Design conclusions and validation framework

The completed architecture study compared two connected operating problems: building the first useful corpus and keeping that corpus current. The conclusions below are design decisions with stated rationale, grounded in documented source behavior, service contracts, and known failure modes—not measured outcomes. The accompanying measures form the acceptance framework for a deployment; they are not presented as results that were never collected.

Initial-ingestion conclusions

Three ingestion patterns were weighed against the same approved multi-system scope:

1. a serial human-assisted producer;
2. one general-purpose agent processing sources sequentially; and
3. the bounded producer fleet with deterministic adapters, Flash-Lite drafting, Flash escalation, validation, and review routing.

The bounded fleet was selected because it preserves deterministic source reads, single-writer publication, and explicit review while still partitioning the inventory for parallel execution. The serial patterns remain useful baselines, but neither provides the same combination of progressive coverage and source-specific concurrency control.

The validation corpus combines database schemas, content-system documents, and an enterprise-application sandbox with deliberately overlapping concepts. Its workload introduces changes during ingestion to verify that progressive generations converge instead of freezing an inconsistent snapshot.

The acceptance measures are time to first useful coverage, time to declared coverage, valid concepts per thousand source objects, source and relationship precision, human review minutes, model cost per accepted concept, duplicate or conflicting concept rate, permission-leakage count, and the value available after each partial generation.

Regeneration conclusions

Three maintenance policies were compared architecturally:

1. periodic or manual refresh;
2. event-driven regeneration without broad reconciliation; and
3. event-driven regeneration with scheduled reconciliation and review gates.

The hybrid policy was selected as the default for important sources. Events provide the shortest path to a targeted refresh, while reconciliation repairs documented gaps such as expired subscriptions, limited replay windows, duplicate delivery, and incomplete change coverage. Event-only operation remains an explicit lower-cost choice for sources whose owners accept its gap risk; scheduled snapshots remain the correct path where reliable events do not exist.

The validation workload introduces schema changes, document revisions, enterprise-object changes, permission revocations, duplicate and out-of-order messages, expired subscriptions, temporary connector failure, and ambiguous supersession. It records source-to-publication lag, change-capture recall, stale-concept exposure, incorrect automatic publication, reconciliation repair yield, human-review load, revocation-to-index-removal time, and recovery after an outage.

Four conclusions follow from the completed analysis:

• C1: the bounded producer fleet is the correct unit of parallelism because it increases concurrency without surrendering deterministic identity, validation, or single-writer publication;
• C2: deterministic extraction paired with a Flash model ladder aligns model capability with task complexity and avoids spending reasoning capacity on facts already available through APIs;
• C3: event-driven regeneration improves timeliness, but reconciliation is the mechanism that restores confidence after event-path gaps; and
• C4: progressive generations are safe to consume before full coverage only when their coverage manifest makes unobserved and failed scope explicit.

These conclusions establish the operating design. The quantitative measures above remain deployment acceptance criteria rather than invented benchmark results. Review load growing linearly with every source change indicates a weak semantic boundary. Reconciliation finding almost no drift supports a lower cadence. Event-only operation missing important changes prevents it from being treated as equivalent to the hybrid policy. A parallel fleet generating more conflicts than the serial baseline shows that work was partitioned at the wrong boundary.

Scope and evidence boundaries

The reviewed Google product documentation does not describe this complete maintenance plane as one managed feature. The OKF reference agent is a proof of concept for BigQuery and approved web enrichment; the multi-source producer fleet is the architecture contribution developed here. Knowledge Catalog can collect and enrich metadata, but an implementation still has to define OKF generation, review, temporal semantics, and source-specific maintenance policy.

Likewise, modification time is not business-valid time. A model can detect that two documents look related, but similarity does not prove that one supersedes the other. An event can reveal that an object changed, but it does not prove that the event stream is complete. Structured output can guarantee shape, not truth. These boundaries are not defects in the design; they are the reason the design keeps deterministic observation, model interpretation, and governed publication separate.

The result is not a knowledge layer that claims to be permanently “finished.” It is a layer that states what it has observed, publishes useful increments, detects when its evidence changes, and regenerates at a cost and risk level chosen for each source. Initial ingestion creates the first trustworthy generation. Maintenance prevents that generation from becoming a confidently stale map.

References

1. Sam McVeety and Amir Hormati, “Introducing the Open Knowledge Format”, Google Cloud, 12 June 2026.
2. GoogleCloudPlatform, Open Knowledge Format v0.2 specification, 2026.
3. GoogleCloudPlatform, OKF reference agent and visualizer, 2026.
4. Google Cloud, Knowledge Catalog overview, 2026.
5. Google Cloud, Gemini 3.5 Flash-Lite, 2026.
6. Google Cloud, Gemini 3.7 Flash, 2026.
7. Google Cloud, Structured output, 2026.
8. Google Cloud, Get batch predictions for Gemini, 2026.
9. Google Cloud, Create Cloud Run jobs, 2026.
10. Google Cloud, Execute Workflow steps in parallel, 2026.
11. Google Cloud, Order Pub/Sub messages, 2026.
12. Google Cloud, Configure Cloud Tasks queue routing, limits, and retries, 2026.
13. Google Cloud, Datastream sources, 2026.
14. Google Workspace, Subscribe to Google Drive events, 2026.
15. Google Drive, Retrieve changes, 2026.
16. Salesforce Developers, Change Data Capture event setup, 2026.
17. Salesforce Developers, Event message durability, 2026.
18. Salesforce Developers, Get object metadata, 2026.
19. Google Cloud, Monitor asset changes with Pub/Sub, 2026.
20. Google Cloud, Cloud Storage request preconditions, 2026.

Suggested citation