Concept Library

A working notebook of how I think about each problem.

50+ entries across data architecture, data engineering, BI and analytics, and agentic AI. Each entry follows the same structure: what it is, why it matters, how it works, vendor comparisons grounded in what I have actually deployed, and a short signed take.

The operating system

How the clusters compose.

01
Data Engineering
Pipelines, compute, & orchestration
02
Data Architecture
Storage, models, & semantics
03
BI & Analytics
Metrics, dashboards, & decisions
04
Agentic AI
Workflows, tools, & orchestration

Data Architecture

14 entries

The shapes data takes when it has to serve multiple consumers reliably — lakes, warehouses, lakehouses, mesh, semantic layers, and the contracts that hold them together.

Browse cluster

Data Engineering

12 entries

The disciplines that make data move reliably at scale — orchestration, transformation, contracts, observability, cost, and the operating model that holds the function together.

Browse cluster

BI & Analytics

8 entries

The layer that turns data into decisions — semantic models, dashboards, decision intelligence, and the editorial discipline of designing metrics for executive use.

Browse cluster

Agentic AI

16 entries

Building AI systems that do, not just answer — agent topologies, guardrails, evaluation, governance, and the architectural patterns that make agentic systems survive production.

Browse cluster

Data Architecture

14 entries

Intermediate

Data Federation and Virtualization

Querying data where it lives across multiple systems through a unified virtual layer, instead of physically copying it into a central warehouse.

Intermediate

Data Lake vs Data Warehouse vs Lakehouse

Three storage and compute paradigms — schema-on-read, schema-on-write, and the convergent model that combines both — each fit for different organizational profiles.

Intermediate

Data Mesh

An organizational and architectural pattern that distributes data ownership to domain teams, treats data as a product, and federates governance — replacing the central data team with a platform-and-policy model.

Intermediate

Data Vault Modeling

An ensemble modeling pattern (hubs, links, satellites) designed for auditability and adaptability in environments with frequent source changes — the right answer for regulated, integration-heavy data warehouses.

Advanced

Lambda vs Kappa vs Unified Streaming

Three architectural patterns for processing batch and streaming data — preserving them as separate paths, collapsing them into one streaming-only path, or unifying them on a single engine.

Intermediate

Medallion Architecture

A three-zone data architecture — bronze, silver, gold — that separates raw, refined, and business-ready data within a single lakehouse.

Foundational

The Modern Data Stack

The cloud-native, modular set of tools — ELT, warehouse, transformation, semantic layer, BI — that replaced monolithic data platforms in the 2015-2020 decade.

Advanced

Multi-Region Data Architectures

Patterns for distributing data across regions to satisfy residency laws, latency requirements, and disaster-recovery commitments — without losing analytical coherence.

Intermediate

OLTP vs OLAP vs HTAP

Three database workload patterns — transactional, analytical, and hybrid — each requiring different storage layouts, query patterns, and operational commitments.

Advanced

Open Table Formats — Iceberg, Delta, Hudi

Open metadata layers built on top of Parquet that bring transactional guarantees, schema evolution, time travel, and indexing to lakehouse storage.

Advanced

Reverse ETL and Operational Analytics

The pattern of pushing modeled data from the warehouse back into operational tools (CRM, marketing, support) — turning the warehouse into the system of record for data that drives action.

Intermediate

The Semantic Layer

A governed contract between physical data and consuming tools — defining metrics, dimensions, and access in one place so that every consumer (BI, ML, agents) reads the same business truth.

Intermediate

Slowly Changing Dimensions

A set of patterns (Type 0 through Type 7) for handling attribute changes in dimension tables — choosing how much history to preserve and how to expose it.

Intermediate

Star, Snowflake, and Galaxy Schemas

Three dimensional modeling patterns that organize fact and dimension tables for analytical consumption — each trading off normalization against query performance and complexity.

Data Engineering

12 entries

Core

Batch vs Streaming vs Micro-Batch

Three processing modes — periodic batches, continuous streams, or small frequent batches — each with different latency, cost, and complexity profiles.

Core

Change Data Capture (CDC)

Capturing every insert, update, and delete from an operational database in near real-time, instead of polling for changes on a schedule.

Core

Cost Optimization for Data Platforms

The discipline of producing the same outcomes for materially less money — through better pipeline design, smarter compute selection, and operational discipline most teams skip.

Intermediate

Data as a Product

Treating each data asset like a product — with an owner, a contract, a roadmap, an SLA, and consumers who can give it the equivalent of an NPS rating.

Core

Data Contracts

An explicit, versioned agreement between data producers and consumers — schema, semantics, SLA, ownership — enforced in CI before bad data reaches production.

Core

Data Quality and Observability

The practice of catching bad data before it reaches a dashboard or a model — through assertions, anomaly detection, and end-to-end lineage.

Core

Data Transformation: dbt, SQLMesh, Coalesce

The transformation layer turns raw warehouse data into modeled, tested, documented tables that the rest of the business consumes. dbt defined the category; SQLMesh and Coalesce are credible challengers.

Intermediate

DataOps Maturity

The practices that make a data platform reliable: version control, CI/CD, environments, automated testing, observability, and clear incident response — borrowed from software engineering and adapted for data.

Intermediate

Idempotency and Exactly-Once Semantics

Designing pipelines so that running the same job twice produces the same result, and so that each input event is reflected in output exactly once — even after retries, failures, and replays.

Core

Orchestration Frameworks: Airflow, Dagster, Prefect, Temporal

The control plane that schedules pipelines, tracks dependencies between tasks, retries failures, and gives engineers a single place to ask 'did the data run, and is it correct?'

Intermediate

Schema Evolution and Backwards Compatibility

How to change the shape of data over time — adding columns, renaming fields, deprecating tables — without breaking the consumers downstream.

Intermediate

Team Topologies for Data and AI

How to organize people around data and AI work — central platform team, embedded analytics engineers, domain ownership, agentic system squads — so the org structure matches the work.

BI & Analytics

8 entries

Intermediate

Anomaly Detection and Forecasting

Statistical and ML techniques for spotting metric movements that warrant attention, and projecting where metrics are likely to land — applied carefully so that signals do not become noise.

Core

Dashboard Design Principles

The visual and structural choices that make a dashboard usable: clear hierarchy, deliberate metric selection, restraint in chart types, and an explicit answer to the question 'what decision does this support?'

Intermediate

Decision Intelligence

Designing analytics around the decision being made — including the action, the decision-maker, and the feedback loop — rather than around the data being available.

Intermediate

Embedded vs Standalone Analytics

The choice between analytics inside the application your customer or employee already uses, versus analytics in a separate BI tool — with very different design, governance, and cost implications.

Intermediate

Executive Metric Design

Selecting, defining, and operationalizing the small set of metrics that drive executive decisions — distinct from the operational metrics teams use day-to-day.

Core

The Modern BI Stack

The architectural pattern of warehouse + semantic layer + BI tool, replacing the monolithic BI platforms of the previous decade.

Intermediate

Reverse ETL and Activation

Pushing modeled data from the warehouse back into operational tools — CRM, marketing automation, support — so that insights become actions in the systems where work happens.

Core

Self-Service Analytics

Letting business users answer their own data questions through governed, modeled, semantic-layer-backed tools — without going through a ticket queue.

Agentic AI

16 entries

Core

Agent Evaluation Frameworks

Systematic ways to measure whether an agentic system works — output evaluation, trace evaluation, regression suites, A/B testing — and the discipline that turns 'it seems to work' into 'we know it works'.

Core

Agentic AI Architecture Patterns

The reusable shapes for organizing LLMs and tools into systems — from single-agent loops to orchestrator-worker hierarchies to graph-based state machines — each with different trade-offs in cost, latency, predictability, and capability.

Intermediate

AI Governance Frameworks

The policies, processes, and roles that govern how AI systems are built, deployed, and monitored — covering risk classification, approval gates, ongoing monitoring, audit trails, and incident response.

Core

Cost and Latency Budgeting for AI

The discipline of designing AI systems within explicit cost-per-run, latency-per-step, and total-budget envelopes — and the architectural choices that make those envelopes hold under real traffic.

Core

Deterministic Guardrails

The non-LLM checks — input validation, output validation, allowlists, budget caps, content filtering — that constrain what an agentic system can do regardless of what the model decides.

Core

Fine-tuning vs RAG vs Tool Use

Three ways to make a model good at your specific problem — teach it (fine-tuning), give it the relevant content at runtime (RAG), or let it act (tool use) — each fitting different problem shapes.

Core

LLM vs Agentic: The Difference

An LLM produces an answer. An agentic system uses LLMs in a loop with tools, memory, and decisions — taking actions, observing outcomes, and continuing until a goal is met.

Core

MCP: Model Context Protocol

An open protocol that lets LLM-powered systems connect to external tools, data sources, and services through a standardized interface — making integrations portable across models, frameworks, and clients.

Core

Memory in Agents

How agentic systems remember — within a single run (working memory), across runs (long-term memory), and across users (shared knowledge) — and what each layer is good and bad at.

Core

Multi-Agent Orchestration

Coordinating multiple specialized agents to solve a problem together — through hierarchical orchestrator-worker patterns, peer-to-peer collaboration, or graph-based state machines.

Core

Production Patterns for Agentic Systems

The architectural and operational patterns that separate agentic systems that survive in production from prototypes that break under load — covering observability, durability, deployment, evaluation, and incident response.

Intermediate

Prompt Engineering vs Architecture

The distinction between writing better prompts and designing better systems — and why most production failures are architectural, not linguistic.

Core

RAG: Retrieval-Augmented Generation

Augmenting LLM context at runtime with relevant documents retrieved from an index — letting models answer questions about content they were never trained on, with citation discipline.

Core

Tool Use and Function Calling

The mechanism that lets an LLM emit structured calls to external functions — query a database, send an email, call an API — turning a text generator into something that can act on the world.

Core

Vector Stores and Embedding Strategy

The storage and retrieval layer for embedding-based search — and the design choices around embedding models, chunking, hybrid retrieval, and operational scale that determine whether RAG works in production.

Core

What Is an Agent, Really

An LLM-powered system that decides what to do next based on its goal, takes actions through tools, observes the results, and continues until the goal is met or the budget is exhausted.

Living document

The library is reviewed quarterly. If a vendor comparison is out of date or a definition has aged, send me a note and I'll update the entry — with credit if you'd like.