Ashutosh Modi
Senior Data Engineer · Jacksonville, FL
I build petabyte-scale data platforms — lakehouse architectures, streaming pipelines, identity resolution at 200M+ scale. U.S. patent holder. I write about data engineering and AI agent systems.
Highlights
- U.S. Patent 12141166B2 — incremental sync framework across Delta Lake and BigQuery; 90% cost reduction.
- Petabyte-scale warehouse — 110B+ records in BigQuery powering self-serve analytics at Sam's Club (Walmart).
- Identity platform — 200M+ identities, 2,000+ attributes on Delta Lake.
- AI thought leadership — three engineering posts featured in Walmart CTO's weekly communication.
Featured work
Streaming Observability Platform with Local LLMs
End-to-end real-time observability: Kafka ingestion, Spark Structured Streaming with tumbling windows and watermarks, rule-based anomaly detection with tiered severity, and local LLM inference for auto-generated incident summaries.
Kafka · Spark Structured Streaming · DuckDB · Ollama
Lakehouse + Streaming Pipeline (in progress)
Building in public: a lean, free-tier lakehouse — real-time API ingestion through Redpanda, Apache Iceberg tables on Cloudflare R2, incremental DuckDB/PyIceberg transforms, Dagster orchestration, and a live dashboard embedded in this site.
Redpanda · Apache Iceberg · Cloudflare R2 · DuckDB · PyIceberg · Dagster