DATA ENG4 days ago
Sumeh: one API for data quality across 14 engines
Every data quality framework eventually forces you into a choice. Great Expectations gives you a rich...
News Feed
60articles · curated from global AI & Data Engineering sources
DATA ENG4 days ago
The Linux Guide I Wish Someone Had Given Me : The 20% you'll use 80% of the time
This is the guide I wish someone had handed me before started learning linux. Enter...
DATA ENG5 days ago
Maternity HL7-to-FHIR Pipeline: Bridging Legacy Hospital Messages to Modern Healthcare APIs
How It Started A few months ago, I was browsing remote FHIR engineer roles in Australia....
DATA ENG7 days ago
Apache Data Lakehouse Weekly: July 11 to July 18, 2026
The open lakehouse stack spent this week arguing about what belongs in a spec and what belongs in a...
DATA ENG7 days ago
Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust
There is an apparent contradiction at the heart of using AI agents for data work, and resolving it...
DATA ENG7 days ago
Architecting lean LLM caching: how to drop a 20M-row table without losing your AI memory
Any team running an agentic pipeline on a periodically-reloaded dataset caches the output of inputs...
DATA ENG9 days ago
LangGraph vs LangChain in 2026: When Each Wins
The data pipelines that power modern finance are getting longer, more dynamic, and more...
DATA ENG9 days ago
I wanted Snowflake to do more than store data. So I built a pipeline where it judges World Cup conviction, then writes its verdict back to Solana. The architecture ended up being my favorite part of the project.
We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to...
DATA ENG11 days ago
The Project That Changed Everything: How Tutoring Juniors on SQL Led Me to Data Engineering
Prefacing A year ago, I would've laughed if you told me this was coming. Back in high...
DATA ENG12 days ago
A Deep Dive Into File Compression: How Data Gets Smaller, Why Codecs Differ, and What to Actually Use in the Lakehouse
Somewhere in your data platform right now, a single configuration property is quietly deciding a...
DATA ENG13 days ago
The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage
For a decade, the file format layer was the most settled real estate in data. Apache Parquet held the...
DATA ENG13 days ago
Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, How They Work, Where They Stand, and Where They're Going
The table format war is over, and the table formats are not. Both halves of that sentence are true,...
DATA ENG15 days ago
When LangGraph Succeeds but Silently Goes Wrong
The most dangerous failure is a clean run The most common mistake teams make when moving a...
DATA ENG16 days ago
Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables
Original Japanese article: Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する ...
DATA ENG16 days ago
3 Staff Engineers Couldn't Pass This Single DE Job Posting
A real 2026 DE job required SQL, Spark, Kafka, Flink, dbt, Terraform, K8s, and LLM skills. Three staff engineers reviewed it. Zero qualified.
DATA ENG17 days ago
Apache Data Lakehouse Weekly: July 1 to July 8, 2026
The lakehouse community spent this week deciding how change itself should work. Apache Parquet opened...
DATA ENG17 days ago
The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade
Apache Parquet is thirteen years old, holds more of the world's analytical data than any other...
DATA ENG17 days ago
The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere
In February 2026, Apache Arrow turned ten years old. The first commit landed on February 5th, 2016,...
DATA ENG17 days ago
The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse
I have a personal stake in this one, so let me declare it up front. Apache Polaris was co-created by...
DATA ENG17 days ago
The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough
The most common architecture question I get in 2026 is no longer "should we use Iceberg." That one is...
DATA ENG17 days ago
The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter
If you want to know where Apache Iceberg is headed, do not read the press releases. Read the dev...
DATA ENG17 days ago
Delete Files vs Deletion Vectors in Apache Iceberg: How V3 Rewrote the Economics of Changing Data
Here is a fact that surprises almost everyone the first time they hear it: in a data lakehouse,...
DATA ENG18 days ago
Meet Apache Ossie: The Open Semantic Interchange Finds Its Home at the ASF
In June 2026, a project quietly entered the Apache Incubator that I believe will matter as much to...
DATA ENG19 days ago
You saved $80K self-hosting Spark. It cost you $400K.
Then he walked me through the team. Two senior engineers, 30% of their time, on Spark cluster...
DATA ENG22 days ago
Data Engineer LeetCode: What to Grind and What to Skip
Most DE candidates grind the wrong LeetCode problems. Here is what actual interview loops test in SQL, Python, and system design in 2026.
DATA ENG23 days ago
Phase 1: Document Ingestion - The Hidden Complexity Before Embeddings
The Story Begins: Why Your Upload Button Is Just The Beginning 👦 Nephew: Uncle! I finally...
DATA ENG24 days ago
[Databricks on AWS #0] The Target Architecture: Isolating Prod, Dev, and Sandbox with Unity Catalog
The north-star design behind our Databricks platform — Lakehouse and Medallion under Unity Catalog governance, three fully isolated zones (prod, dev, sandbox), zero-copy reads, and a promote path that keeps humans out of production writes.
DATA ENG24 days ago
Apache Data Lakehouse Weekly: June 24 to July 1, 2026
The open lakehouse runs on a small stack of Apache projects, and this was a week where those projects...
DATA ENG25 days ago
DE Interviews Dropped DSA. The Replacement Is a Mess.
Companies quietly killed LeetCode rounds for DE roles , but what replaced them is inconsistent, unpredictable, and harder to prep for than algorithms ever were.
DATA ENG26 days ago
I Built an Open Healthcare Data Dictionary with 100k+ ISO-11179 Terms — Here's What I Learned
If you've ever worked on a healthcare data team, you've lived this problem: one system calls it...
DATA ENG26 days ago
The customer_id that isn't a customer
An error is the safe outcome. The dangerous one is a number. When a text-to-SQL feature breaks the...
DATA ENG26 days ago
Two audits of my own knowledge graph found two unrelated silent failures
I'm a systems administrator in the Dominican Republic, and for the past several months I've been...
DATA ENG27 days ago
How to Build a Modern On-Premise Data Lakehouse (Without Vendor Lock-in)
Building a modern analytical platform usually rhymes with one thing: migrating to the cloud. But...
DATA ENG27 days ago
I Built a Free Apache Kafka Course from Scratch — Here's the Full Curriculum (and What I Got Wrong)
I Built a Free Apache Kafka Course from Scratch — Here's the Full Curriculum (and What I Got...
DATA ENG28 days ago
Why *= 2 is slower than = col * 2 in Pandas — and what strace shows about abstraction cost
This problem looks trivial, but I got curious about what different Pandas update strategies actually...
DATA ENGabout 1 month ago
Less Database, More Files
A few things I noticed. None of them alarming on their own. Our database doubles in size every...
DATA ENGabout 1 month ago
Your Data Engineering Take-Home Is Now 20 Hours of Free Work
Take-homes grew from 4 hours to 20. No pay, no feedback, AI banned with no rubric updates. The DE interview is now just unpaid consulting.
DATA ENGabout 1 month ago
Snowflake vs Databricks, BigQuery vs Redshift? The 2026 Guide to Right-Sizing Your Data Platform
TL;DR Big data platforms like Snowflake and BigQuery impose high pricing floors, like...
DATA ENGabout 1 month ago
How Uber Built Its Big Data System — From a Few TBs to 350 Petabytes with Sub-Hour Latency
A deep-dive into Uber's three architectural generations — from a Vertica warehouse to a Hadoop data lake, to Apache Hudi — and the engineering decisions behind 6 trillion rows ingested daily.
DATA ENGabout 1 month ago
I'm opening ContractForge — define data ingestion intent once, run it natively anywhere
Every data engineer who works across platforms knows this pain: You build a clean ingestion layer...
DATA ENGabout 1 month ago
Data Contracts in Production: Stop Trusting Your Upstream Sources
Your upstream data source changed a column type last night. Your pipeline ran at 2am, ingested...
DATA ENGabout 1 month ago
Synthetic Data for Data Engineering: How to test a Pipeline before the real data arrives
There is a quiet absurdity at the center of most data work, and once you notice it you cannot stop...
DATA ENGabout 1 month ago
Unlock Kafka Schemas with Karapace: A Hands-On Guide
TL;DR Kafka handles event streaming: producers write to topics, consumers read. JSON...
DATA ENGabout 1 month ago
Part 1 — How Do Unstructured Documents Become a Searchable Knowledge Base? Five Key Engineering Decisions in the Ingestion Pipeline
This article covers the first layer of the full-stack architecture: the Ingestion Pipeline. If this...
DATA ENGabout 1 month ago
Top 12 Spark Interview Problems for Data Engineers, With Answers
12 real Spark interview problems on shuffle, skew, caching, AQE, window functions, and OOM debugging, with detailed answers.
DATA ENGabout 1 month ago
I Built a Mini Message Broker in Pure Python and Finally Understood How Kafka Moves Millions of Events
Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag...
DATA ENGabout 1 month ago
I Built a B-Tree in Pure Python and Finally Understood Why Postgres Uses It for Every Index
I Built a B-Tree in Pure Python and Finally Understood Why Postgres Uses It for Every...
DATA ENGabout 1 month ago
I Built a World Cup Prediction Model That Retrains Itself Daily and Can't Cheat Its Own Results ⚽
Most sports prediction models have perfect hindsight. Mine is committed to git before...
DATA ENGabout 1 month ago
I Built a Token-Bucket Rate Limiter in Pure Python and Finally Understood How APIs Protect Themselves
Every production API does it. Stripe does it. Twilio does it. Your own data pipeline endpoints...
DATA ENGabout 1 month ago
I Built a Streaming Window Aggregator in Pure Python and Finally Understood How Flink Handles Late Data
I spent two years writing Flink jobs before I understood what a watermark actually does. Not what...
DATA ENGabout 1 month ago
Apache Data Lakehouse Weekly: June 4 to June 11, 2026
The lakehouse community spent this week arguing about versions, and the arguments mattered. Parquet...
DATA ENGabout 2 months ago
The Search Engine Renaissance: How Apache Lucene and Elasticsearch Are Reclaiming the AI-Native Future
"The reports of my death are greatly exaggerated." — Mark Twain, if he were a search engine. For a...
DATA ENGabout 2 months ago
I built a mini columnar storage engine in pure Python — and finally understood why Parquet is so fast
Last month a "fast" reporting query at work was taking 40 seconds. The table had 90 columns. The...
DATA ENGabout 2 months ago
Your Scraper Returns 200 OK and Lies. Here's How to Catch It.
For a decade, scraping defense was about blocking. Status codes, IP bans, captchas. You knew when you...
DATA ENGabout 2 months ago
I built a Databricks medallion lakehouse to roast my own YouTube history (Bronze Silver Gold Existential Dread)
There's a normal way to analyze your YouTube watch history. You export it from Google Takeout, open a...
DATA ENGabout 2 months ago
Apache Data Lakehouse Weekly: May 28 - June 4, 2026
The lakehouse projects spent this week doing two things at once. They lined up a remarkable stack of...
DATA ENGabout 2 months ago
FastAPI for Data Engineers: Building, Testing, and Debugging APIs That Don't Lie to You
The JobSense project needed a FastAPI backend that served 604 job embeddings via semantic search, a...
DATA ENGabout 2 months ago
The Data Engineering Take-Home Assessment: How to Turn a 4-Hour Test Into a Job Offer
Most candidates treat the take-home assessment as a coding test. It is not. It is a professional...
DATA ENGabout 2 months ago
Data Engineering Interviews with No Industry Experience: A Playbook That Actually Works
I graduated in November 2025. My only formal work experience is a 6-month IT internship. I have never...
DATA ENGabout 2 months ago
SQL Interview Problems for Data Engineers: 30 Patterns That Actually Get Asked
Data engineering SQL interviews are not the same as software engineering SQL interviews. You will not...