DATA ENGabout 16 hours ago
Deploying Apache Flink on a Kubernetes Cluster as an Alternative to GCP Dataflow
Google Cloud Dataflow is a managed stream and batch processing service built on Apache Beam that...
News Feed
60articles · curated from global AI & Data Engineering sources
DATA ENGabout 17 hours ago
ELT with Dataform on Google Cloud
A real-world ELT pipeline with Dataform and BigQuery — staging and mart layers, JavaScript includes, dynamic tables and views, GitHub sync over SSH and Terraform provisioning.
DATA ENG1 day ago
Four Iceberg Tools, Three Agent Frameworks: What Ports, and What Doesn't
Four read-only Apache Iceberg tools bound into Google ADK, AWS Strands and Microsoft Agent Framework, run against five catalogs, 360 timed runs. Building the agent ports and running it does not; speed follows the model and how much it writes; the storage wiring under the tools is the per-cloud work.
DATA ENG2 days ago
One Table, Many Engines: Running BigQuery, Spark, and Trino on a Single Iceberg Copy
A hands-on guide to building an open lakehouse on Google Cloud: one Apache Iceberg table in GCS, three engines, real OCC conflict (HTTP 409) via the Lakehouse Iceberg REST Catalog.
DATA ENG2 days ago
Is SQLazy Something Missing from Your dbt Workflow?
dbt is good – but it has blind spots If you’re working in analytics engineering, dbt is...
DATA ENG4 days ago
56 fault-injection tests passed. The one that injected nothing failed.
I was building a tool that detects when data quietly changes meaning — a vendor switching units, a...
DATA ENG5 days ago
Window Functions Are the SQL Superpower Nobody Taught You
Running totals, top-N-per-group, and row comparisons — without the self-join headache. A practical walkthrough with real code and a cheat sheet.
DATA ENG6 days ago
I Built a Data Quality Library Because Existing Tools Were Too Slow for My Pipelines
I got mass fed up with data quality tools, so I built my own I work as a data engineer....
DATA ENG7 days ago
Apache Data Lakehouse Weekly: September 3-9, 2026
Three things ran through the dev lists this week. Projects kept redrawing their own boundaries, with...
DATA ENG7 days ago
The Hard Truth About Data Engineering Interviews Nobody Tells You
LeetCode won't save you. SQL will. Here's what data engineering interviews actually test, with real examples.
DATA ENG8 days ago
5 Habits That Keep a dbt Project From Turning Into a Mess
I once inherited a dbt project where the same "active customer" filter existed in four different...
DATA ENG10 days ago
Snowflake Multi-Environment Account Architecture: RBAC, Cost Governance, Identity Isolation & PII Masking Design
Designing account-level isolation for a multi-environment Snowflake account: one environment, one...
DATA ENG12 days ago
The hardest label in our dataset is the one where nothing happened
Four times a year, a public company gets on a call with analysts and says what it intends to do next....
DATA ENG13 days ago
AWS Has Two Iceberg REST Catalogs: What Each One Actually Serves
Glue and S3 Tables both implement the Apache Iceberg REST catalog specification. One identical request suite against both shows the same totals and thirteen behavioural differences, including two that require opposite things of the same drop request.
DATA ENG13 days ago
Seven Iceberg REST Catalogs: What They Declare, and What They Serve
One request suite against seven Apache Iceberg REST catalog implementations — Polaris, BigLake, Glue, S3 Tables, Unity, Horizon and OneLake — comparing what each catalog declares it supports against what it actually serves.
DATA ENG13 days ago
I Analyzed 2,061 Public n8n Workflows and Found 92% of the AI Nodes Had Been Silently Replaced
TL;DR: Tried to compare AI-agent n8n workflows against plain ones. Filtering the biggest public...
DATA ENG13 days ago
I built a hockey analytics pipeline that scrapes, models, and publishes itself every week — no server, no database
Kenya Hockey Union publishes match results on its website. That's it — no API, no downloadable...
DATA ENG13 days ago
Azure Synapse Databricks: A Migration Playbook That Survives Production
Azure Synapse → Databricks: A Migration Playbook That Survives Production A field-style...
DATA ENG14 days ago
The Junior Data Engineer Job Is Gone. AI Took It.
Junior DE postings collapsed 67%. AI does the work they trained on. Companies still hire, but only senior. The entry point into DE is gone.
DATA ENG16 days ago
Data Engineering Dropped DSA. What Replaced It Is Worse.
Companies killed algorithm rounds but replaced them with 4 incompatible formats, AI bans 80% ignore, and no rules anyone agrees on. Here is the chaos map.
DATA ENG17 days ago
Building an investing knowledge graph, part 3: same company, seven names
When entity resolution fails, the same company becomes seven different graph nodes. What that looks like in practice, using Samsung Electronics, Samsung SDI, and ASML.
DATA ENG20 days ago
ClickHouse 26.8 LTS: 57 Breaking Changes Since 26.3
If you run ClickHouse in production, you're probably on 26.3 LTS. And now 26.8 LTS has been...
DATA ENG20 days ago
Apache Spark & PySpark, Explained Without a PhD
The first time I hit big data I didn't even know that's what it was.
DATA ENG21 days ago
Data Engineers Now Out-Earn Data Scientists. AI Engineers Beat Both.
The salary hierarchy just flipped. Senior DEs clear $179K base. AI engineers hit $280K TC. Here is what changed and what it means for your title.
DATA ENG21 days ago
Stop Re-Receiving the Advice You Already Rejected
Third time in the same thread it suggests change data capture. You ruled that out an hour ago, and you never said so out loud.
DATA ENG23 days ago
Your Take-Home Was Free Labor. The Data Proves It.
DE take-homes now run 20 hours. Senior engineers fail new-grad screens. Loops last 90 days. The interview is broken in a new, expensive way.
DATA ENG23 days ago
Technically Right, Organisationally Impossible
The architecture is correct. You cannot build it. Not for technical reasons, but because of who would have to be awake at 3am to run it.
DATA ENG24 days ago
Teradata, Explained Like the Warehouse That Was There First
Every large, established company seems to have one: a big, powerful data system
DATA ENG25 days ago
Claude Assumed You Were Building Greenfield. You Were Not.
The recommendation is clean, modern and sensible. It also assumes you do not have an Airflow instance, four hundred stored procedures, and a team that just learned dbt.
DATA ENG27 days ago
Understanding the ORC File Format: Why Is It So Fast
Every article about ORC says the same thing. "Columnar storage, built-in indexes, predicate pushdown...
DATA ENG27 days ago
You Are Describing the Symptom, Not the Problem
You ask why there are nulls in the report and get good advice about handling nulls. The nulls were never the problem.
DATA ENG28 days ago
61% of AI Cheaters Get Offers. Honest Engineers Lose.
48% of candidates use AI in interviews despite bans. 61% of them get offers anyway. The cost lands on the engineers who follow the rules.
DATA ENG29 days ago
Data Lakehouse: How and Why
How the data lakehouse works in production — the internal mechanics, the data flow patterns, why...
DATA ENG29 days ago
Can We Stop Running Recurring Analysis as a Scheduled LLM Prompt?
Every major assistant now ships some version of scheduled prompts. You write "every Monday at 9am,...
DATA ENG30 days ago
48% of Data Engineering Job Postings Are Ghost Jobs
Nearly half of DE job postings may never be filled. Candidates spend 20 hours on take-homes for roles that aren't real. Here is what is actually happening.
DATA ENGabout 1 month ago
Databricks Changed One Permission Rule. Grants You Forgot About Are Now Live.
On August 3, 2026, a short entry landed in the Azure Databricks release notes. No keynote, no blog...
DATA ENGabout 1 month ago
Ask the Right Questions First
The questions to ask before starting a data migration: what success looks like, who signed the contract, who decides on data, what the recovery plan is, and what happens to work created during the migration.
DATA ENGabout 1 month ago
interlace.sh: Why We Built a Unified Abstraction
Consider what a data engineer has to learn before writing a single useful transformation on a typical...
DATA ENGabout 1 month ago
The One-Minute Trap: What Microsoft Fabric's New Warehouse Billing Model Means for Your Workloads By Gilbert Kiptoo Lelon | DP-700|DP-600
Microsoft is changing how Fabric Data Warehouse charges for compute starting August 2026. Most teams...
DATA ENGabout 1 month ago
The Airbyte Column Nobody Queries: Find Every Silent Sync Failure in 15 Minutes
You added a new field to your CRM. Three weeks later, someone in finance asks why 4% of deal values...
DATA ENGabout 1 month ago
IBM Bought Confluent for $11B. Then Fired 800 Engineers.
The biggest DE layoff of 2026 just flooded your job market. What the Confluent massacre means for Kafka careers, streaming skills, and where the 800 land.
DATA ENGabout 1 month ago
The First Answer Is a Starting Position, Not a Verdict
You read the answer, decide it is about 80% right, and fix the rest yourself. There is a better second move, and also a worse one that most people reach for.
DATA ENGabout 1 month ago
Looker Pricing in 2026: What Google Publishes, What You Actually Pay
Looker pricing 2026: no published list prices, but precise Gemini Data Token rates. What buyers actually pay and the LookML labour line explained.
DATA ENGabout 1 month ago
You Review AI Code Less Carefully Than Human Code. That Is Backwards.
Not on day one. Day one you are careful. The problem is what happens to your attention after the two hundredth AI pull request.
DATA ENGabout 1 month ago
I built a real Spark event-log parser instead of faking it with sample data
Spark's rules were built and tested but ran on sample data — here's the real event-log parser that replaced that, and the CPU-utilization bug it exposed.
DATA ENGabout 1 month ago
Feed Your AI Agents Clean Data
Most AI projects run into the same problem: a capable model, a motivated team, and a data set that...
DATA ENGabout 1 month ago
dbt Semantic Layer vs Cube vs AtScale: Choosing an Enterprise Semantic Layer
dbt Semantic Layer vs Cube vs AtScale: authoring, pricing, constraints, and AI readiness compared for enterprise evaluators.
DATA ENGabout 1 month ago
The Real-Time Fetish: Why You (Probably) Don't Need Streaming
In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business...
DATA ENGabout 1 month ago
Why F1 Is the Wrong Objective for Entity Resolution Thresholds in Corporate Data
Most entity resolution pipelines are tuned to maximize F1. For corporate data, that is the wrong...
DATA ENGabout 1 month ago
One Bug Hit 11 Rows. Another Hit 36 Million.
I built a predictive maintenance model on 37,164,054 telemetry records from an iron ore mine. Six...
DATA ENGabout 1 month ago
You Forgot to Mention the 500 Million Rows
The query is genuinely elegant. It's also a self-join across a table you never mentioned has half a billion rows. It ran fine against your dev sample.
DATA ENGabout 1 month ago
Storix: One Typed Python Filesystem API from Local Development to Cloud Storage
Storix is a typed sync and async filesystem API for Python. It gives application code one set of...
DATA ENGabout 2 months ago
Monitoring Airbyte's Postgres CDC Replication Slots Before They Take Down Your Source Database
Every Postgres CDC source works the same way under the hood: it opens a logical replication slot, and...
DATA ENGabout 2 months ago
Building Airbyte Connectors That Survive Flaky APIs: Cursor Checkpointing for Resilient Incremental Sync
Most Airbyte custom-connector tutorials assume the happy path: the upstream API always responds,...
DATA ENGabout 2 months ago
Apache Iceberg Query Performance: A Practical Guide
Iceberg is the standard. Snowflake, Databricks, AWS, and every major query engine read and write it...
DATA ENGabout 2 months ago
Row and Field-Level Data Provenance: Why It's Worth the Pain (and Where the Pain Is)
Most "data lineage" you've seen answers a schema question: table B comes from table A, or column...
DATA ENGabout 2 months ago
Testing a data pipeline against the spreadsheets it replaced
Replacing a reporting workbook means first proving you match it. Then the diffs start pointing the other way, and the incumbent is the broken one.
DATA ENGabout 2 months ago
Entry-Level Data Engineering Is Gone. Here's the Proof.
Junior DE postings collapsed 67%. Only 3% of jobs are entry-level. Hiring is up , but every gain went to senior roles. The on-ramp closed.
DATA ENGabout 2 months ago
You Deleted BM25 the Day You Bought a Vector Database. Your Fraud Queue Noticed
BM25 is a ranking function from the 90s. No GPU, no embedding API, no per-token bill. And in a fraud...
DATA ENGabout 2 months ago
Your scraper isn't broken. The site changed, and it didn't tell you.
The scraper had been green for eleven weeks. Every run exited 0, every run wrote rows to the...