DATA ENG2 days ago
Snowflake vs BigQuery vs Redshift: How to Compare Them
Snowflake vs BigQuery vs Redshift side by side: architecture, pricing, performance tuning, concurrency, Iceberg and governance, and which to pick when.
News Feed
60articles · curated from global AI & Data Engineering sources
DATA ENG4 days ago
I got put in token quota jail. Then I looked at the logs.
Recently, Claude Code just... stopped. Not a bug in my code, not an outage. I had burned through my...
DATA ENG5 days ago
Your operators logged 3 in 10 stops. Your machines caught 7.
What you'll get out of this In about 15 minutes, you'll walk away with: Why your...
DATA ENG6 days ago
Switch from zero: ship a human-in-the-loop data quality agent to your team chat
I spent the Flip the Switch hackathon building Shield, an agent that checks a daily table, explains...
DATA ENG6 days ago
Your Dashboard Is Lying: The Spend Fan-Out Trap and the Lead I Lost When the API Went Down
The Problem That Costs Real Budget Marketing attribution is broken. Not because the math...
DATA ENG6 days ago
Your Iceberg tables are rotting: we measured it, then built a fix (free, Apache 2.0)
We put the scheduled Spark compaction job that most Iceberg teams write on a table taking a MERGE...
DATA ENG7 days ago
Qual a melhor arquitetura de dados para a sua aplicação de IA?
Uma análise de Warehouse, Lake e Lakehouse e dos requisitos que cada um atende em ML, deep learning e RAG.
DATA ENG9 days ago
Bridging the Lakehouse: Ingesting Databricks Unity Catalog into Cognee Knowledge Graphs
How to connect Databricks Unity Catalog data governance, Delta tables, schemas, and AI models into Cognee GraphRAG for conversational data discovery.
DATA ENG10 days ago
The last month is always a lie: lessons from querying 1 billion rows of Brazilian crime data
We run Crime Brasil, a site that turns official Brazilian public-safety data into maps and rankings...
DATA ENG13 days ago
Delta Lake Schema Evolution and Time Travel on Azure Databricks: mergeSchema, overwriteSchema, Column Mapping, and RESTORE
Sooner or later a source system adds a column, renames one, or someone runs a bad overwrite on a...
DATA ENG13 days ago
Data apps, defined: levels, principles, and a scorecard builders can actually use
The Data App Framework, v2.0 · October 2026 Venkata M Sangaraju, Sr. Database Engineer On October 8,...
DATA ENG14 days ago
Your ERP extract is 31% wrong. Nothing errored.
What you'll get out of this In about 12 minutes, you'll walk away with: Why the obvious...
DATA ENG15 days ago
Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES
On September 1, 2026, Databricks made automatic change data feed (Auto CDF) generally available. The...
DATA ENG15 days ago
Stop Shipping AI in Pieces: Why Databricks Is the One Place Where Data, AI, and Apps Finally Live Together
Your AI app is not slow because your model is weak. It is slow because your data, your model, and...
DATA ENG16 days ago
PySpark Window Functions on Databricks: Latest-Record Dedup, Running Totals, Sessionisation, and the Frame Traps
Window functions show up in almost every PySpark interview, and in almost every real pipeline:...
DATA ENG17 days ago
Spark Join Strategies on Databricks: Broadcast, Sort-Merge, Shuffle Hash, and What AQE Changes
Most slow PySpark jobs I've debugged weren't slow because of bad code. They were slow because Spark...
DATA ENG18 days ago
The demo works. Production doesn't: what breaks when you put an LLM on your company's data
Every conversational analytics demo follows the same script. Someone connects a model to a clean...
DATA ENG19 days ago
DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap
DuckDB 2.0.0 shipped on October 21, 2026 (announcement, PyPI). The next morning I ran the same...
DATA ENG19 days ago
What Breaks When You Migrate Petabytes: A Field Guide
By Prajakta Shende — Database Customer Engineer at Google Cloud, with close to a decade at AWS in the...
DATA ENG19 days ago
HubSpot'tan Salesforce'a veri taşırken External ID, upsert ve ilişki sırası: Geliştirici notları
CRM göçü dendiğinde çoğu ekip önce lisans fiyatlarını, sonra arayüzü konuşur. Oysa projeyi gerçekten...
DATA ENG19 days ago
What AI Agents Taught Me About Building Databricks Apps
What AI Agents Taught Me About Building Databricks Apps The agents wrote most of...
DATA ENG19 days ago
Six things that break when a data microservice goes to production
Auth, schema validation, background jobs, Docker shutdown, and the Polars quantile trap. What actually hurts when a notebook pipeline becomes an API.
DATA ENG19 days ago
Auto Loader and Structured Streaming on Azure Databricks: Checkpoints, Schema Evolution, and Exactly-Once
Almost every Azure Databricks project I've seen starts the same way: files land in ADLS Gen2, and...
DATA ENG20 days ago
Writing a Reconciliation Job in SQL and Python That Proves Two Systems Agree
On one engagement, operations and finance relied on a SQL Server data warehouse that brought...
DATA ENG21 days ago
When ClickHouse merges change your query results
In Part 1 I followed an order from INSERT through a merge. That answered two questions that came up...
DATA ENG21 days ago
Parsing credit card statement PDFs: signs, detail lines and a total that has to tie
When you extract transactions from credit card statement PDFs, finding the table is rarely the hard...
DATA ENG21 days ago
Why Stripe payouts don't match your books
A synthetic Stripe example: a €3,496.99 gap between ledger and bank, explained level by level with SQL.
DATA ENG22 days ago
Mastering Intermediate & Advanced SQL: A Practical Healthcare Analytics Guide
In clinical operations and healthcare administration, raw relational tables alone cannot answer...
DATA ENG23 days ago
Follow one row: what ClickHouse does with your INSERT
I work as a data infrastructure engineer at a company where ClickHouse comes up at almost every...
DATA ENG23 days ago
"What breaks if this column changes?" Building LineageGate on FalkorDB
By Alex Suarez · written for the Graph Hacks: Context for AI Agents hackathon (WeMakeDevs ×...
DATA ENG25 days ago
Parsing QFX files: an annotated sample, FITID dedupe, and the pitfalls that bite
A QFX file is OFX with Intuit's INTU.BID tag added, served by banks for Quicken's Web Connect import....
DATA ENG26 days ago
Building Fault-Tolerant CSV Ingestion Pipelines in Python: Handling Mixed Dtypes, Memory Spikes, and Silent Schema Drift
CSV files are the cockroach of software engineering: unstandardized, fragile, lacking embedded...
DATA ENG27 days ago
Athena read my timestamps as UTC. It didn’t tell me.
Nobody gets an error. Somebody gets blamed. I built a fake metal fab plant to find this kind of bug...
DATA ENG28 days ago
Polars 2.0: five things that change when you upgrade, three of them silently
Polars 2.0.0 shipped on 6 October. Most of its breaking changes are loud: a removed argument raises...
DATA ENG28 days ago
5 SQL window-function traps that show up in data engineering interviews (with runnable DuckDB examples)
Window functions are the most common SQL topic in data engineering interviews, and most candidates...
DATA ENG29 days ago
An AI-Assisted Development Methodology, Earned from Real Incidents
This document is a methodology for building software with AI assistance, compiled from the rules that...
DATA ENG29 days ago
Python 3.15 Is Not a Speed Release for Data Teams. It Is an Operations Release.
Python 3.15.0 shipped on October 9, 2026 (release post). The headline number most people will repeat...
DATA ENG29 days ago
AWS Glue Concurrency Is Capped by One Subnet, Not Your Whole VPC
TL;DR If you run many AWS Glue jobs concurrently inside a VPC, your concurrency ceiling is...
DATA ENGabout 1 month ago
SQL Indexes Explained (For People Who Just Got Yelled At By a Slow Dashboard)
At some point in every junior developer's career, a query that ran fine on your laptop turns into a...
DATA ENGabout 1 month ago
My RAG pipeline died three nights in a row. So I rebuilt it to assume it would die.
After my document-ingestion pipeline died three nights in a row, I stopped trusting "it should work"...
DATA ENGabout 1 month ago
Immutable Data ClickHouse, Mutable Data StarRocks: Why I Moved My CDC Tables
TL;DR — ClickHouse is still the tool I reach for when data is written once and never changes:...
DATA ENGabout 1 month ago
Batch ETL vs Real-Time ETL Tools: Which One Your Team Needs
Quick answer: Sort your consumers by how stale their data can be. If every one of them tolerates...
DATA ENGabout 1 month ago
Fan-Out Joins: The Silent Cause of Inflated Metrics
A non-unique join key multiplies facts and inflates aggregates. How to detect and prevent fan-out.
DATA ENGabout 1 month ago
Fivetran vs Estuary for Real-Time ETL
Quick answer: Estuary is the better tool for real-time ETL when a consumer needs database changes in...
DATA ENGabout 1 month ago
I scanned 1,152 dbt models for joins that inflate SUM. Here is what I found.
There is a bug in analytics SQL that never throws an error. select o.customer_id, sum(o.amount) as...
DATA ENGabout 1 month ago
Your data engineer job description is trying to hire a team
A recent r/dataengineering discussion started with a frustrating result. A company interviewed 22...
DATA ENGabout 1 month ago
A cache key ate 99.9% of my records and the pipeline looked green
The failure mode nobody alerts on: a pipeline that succeeds, completes on time, and quietly writes almost nothing. Here's how it happens and how to make it impossible.
DATA ENGabout 1 month ago
How AI can investigate a 12 GB export without reading it into a prompt
Imagine asking an AI assistant for the average order value in a 12 GB sales export. It finds an...
DATA ENGabout 1 month ago
How I Built a Two-Tier Data Platform in 9 Phases (Spark, Kafka, dbt, Kubernetes)
A complete data platform — batch + streaming, single-node + distributed, local + Kubernetes. What I built, what broke, and what I learned.
DATA ENGabout 1 month ago
The fastest Glue job is the one you never start: removing per-job overhead from a 191-job pipeline
In my last post I covered making a production AWS Glue pipeline run faster by tuning individual jobs....
DATA ENGabout 1 month ago
A unique + not_null suite stopped 3 of 17 bad batches. Here is what got through.
Many dbt projects start with the same two tests on every key: unique and not_null. They are cheap,...
DATA ENGabout 1 month ago
I turned Brazil's 73M-establishment company registry into second-fast lookups, with no server and no database
Brazil's tax authority (Receita Federal) publishes its entire company registry (CNPJ) as open data...
DATA ENGabout 1 month ago
How to Build an Open Lakehouse on Your Laptop
An open lakehouse is one where every layer (storage, table format, engine, catalog, and the ML and AI...
DATA ENGabout 1 month ago
What Actually Happens When You Call spark.read? One Line of Python, a Thousand Tasks
TL;DR: spark.read.parquet(path) reads almost nothing. The real work starts at the first action, when...
DATA ENGabout 1 month ago
A bad batch loaded with zero errors. One row added $4.5 million.
I built a small warehouse, fed it a batch of orders with twelve defects planted in it, and watched...
DATA ENGabout 1 month ago
The Distributed Compute Playground: Everything You Can Plug In and Tune in an Apache Spark Job
TL;DR: An Apache Spark job is not one thing. It is a stack of nine layers, and almost every layer...
DATA ENGabout 1 month ago
Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew
Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a...
DATA ENGabout 1 month ago
Your dashboard is green and the number is wrong: the SQL checks I schedule next to every metric
Most broken dashboards don't look broken. The chart renders, the numbers are plausible, the pipeline...
DATA ENGabout 2 months ago
What One Rust Client Can Reach Across Seven Iceberg Catalogs
Step by step: the Apache Rust Iceberg REST client against seven Iceberg catalogs. It logs in to five of them, and on every one it logs in to, all 13 endpoints it implements answer — 7 reads on three catalogs and 11 writes on the local one, with no failures. The two AWS catalogs need SigV4, which it cannot send, so Rust reaches them through two other crates whose operation sets differ from its own.
DATA ENGabout 2 months ago
AWS Glue Iceberg Optimizer Alternatives for a Data Lakehouse
The AWS Glue Data Catalog's managed table optimizers are the path of least resistance for Iceberg...