Methodology & Sources

Sourcing Criteria

Every document included in an Anvil Index pack meets a rigorous set of sourcing standards. This ensures that the chunks you ingest into your vector database are authoritative, current, and legally sound.

Government and Institutional Sources Only

We select documents from federal agencies, standards bodies, and international regulatory organizations. This includes the FDA, NIST, SEC, EU Parliament, EMA, OWASP, GAO, and peer bodies. We do not include blog posts, vendor commentary, or secondary analysis.

Publicly Available

All source documents must be freely accessible without paywalls or subscriptions. Regulatory guidance must be published by the issuing authority and available for unrestricted download or web access.

Authoritative and Official

Sources must be official guidance, binding regulations, established frameworks, or published recommendations from the issuing body. We prioritize binding regulatory text over non-binding guidance, though both are included when relevant to the domain.

Primary Sources Over Secondary

We source from original documents: the actual regulation, framework, or guidance issued by the regulatory body. We do not create packs from interpretations, summaries, or analyses of those documents.

Licensing Verification

Every source document is licensed for distribution. This includes public domain documents, Creative Commons licensed materials with appropriate attribution, and materials explicitly published under open distribution by the issuing authority. The OWASP Top 10 for LLM Applications is dual-licensed under CC BY-SA 4.0, for example, and is included with full attribution metadata.

Chunking Methodology

The integrity of a RAG pack depends on how source documents are segmented into chunks. Our approach balances atomic semantic completeness with practical token constraints for embedding and retrieval.

Atomic Information Principle

Each chunk contains one self-contained concept, requirement, or rule. A chunk on data minimization, for example, stands alone and can be understood without reading surrounding sections. This maximizes the chunk's utility in vector search and reduces noise when the chunk is retrieved alongside other results.

Token Ceiling and Segmentation

Chunks are capped at approximately 1,500 tokens to ensure compatibility with common embedding models (OpenAI, Claude, Cohere, Ollama) and to keep context windows manageable in retrieval workflows. In practice, the average chunk across all packs is 400 to 500 tokens, providing ample detail while maintaining precision.

Natural Document Boundaries

Chunks follow the structure of the source document: sections, articles, subsections, and individual requirements are natural breaking points. This preserves the document's logic and makes it easy to trace a chunk back to its source.

Self-Sufficient Context

Each chunk preserves enough surrounding context to stand alone. References to related sections are included as explicit cross-references in the metadata. A chunk on AI auditing requirements will include enough detail about the entity being audited and the audit scope, even if the source document defines those terms elsewhere.

Dual Format Delivery

Every chunk is delivered in two formats. Markdown (.md) is human-readable, suitable for review and documentation. JSON (.json) includes full metadata and is optimized for programmatic ingestion into vector databases and search systems.

Schema Walkthrough

Each chunk carries 17+ metadata fields that enable hybrid search, faceted filtering, and full traceability to the source. This structured metadata is what makes RAG packs more powerful than raw documents.

Hybrid Search Capability

The schema enables two simultaneous search strategies. Semantic search finds chunks by meaning and similarity in the vector space. Metadata filtering narrows results by domain, source authority, jurisdiction, and other structured fields. Together, they allow you to ask questions like: "Find all security requirements from NIST that apply to LLMs" or "List all EU privacy obligations for customer data in FinServ."

Controlled vocabularies ensure consistent filtering across all chunks. All sources, authorities, entities, and obligations use the same standardized terminology, making faceted search reliable and predictable.

Source Index

All 44 source documents organized by pack and domain. Each entry includes the document name, issuing authority, and chunk count.

Versioning & Updates

Anvil Index packs are living products. We monitor source documents monthly and release updates as regulatory guidance evolves.

Semantic Versioning

Each pack uses semantic versioning (MAJOR.MINOR.PATCH). Major versions indicate significant structural changes or substantial new source additions. Minor versions add new chunks or sources. Patch versions correct errors or improve metadata without adding new content.

Change Documentation

Every pack includes a CHANGELOG documenting all changes between versions. You can see exactly what changed, which chunks were added or modified, and when.

Monthly Monitoring

We review all source documents monthly for updates, amendments, and new releases. When a source document is updated, the affected chunks are regenerated with current content and refreshed metadata.

New Sources and Cross-References

New source documents are evaluated against our sourcing criteria. When a new document meets the threshold for inclusion, it is added to the appropriate pack. Cross-references are maintained across packs, so updates in one pack may trigger crosswalk updates in others.

Update Notifications

Customers receive update notifications for packs they have purchased. You can review the changelog and decide whether to upgrade.