15+ hour, 18+ min ago  (561+ words) Which side are you on: "This is just what Kafka costs at scale" or "We should switch to a cheaper Kafka provider"? At Conduktor, our field team works inside Kafka environments that have been running for a long time. We…...

1+ day, 16+ hour ago  (16+ words) Introduction Apache Kafka and Apache Cassandra pair effectively because they complement. .. Tagged with dataengineering, cassandra, kafka....

1+ day, 12+ hour ago  (61+ words) Introduction Ever wondered how banks are able to detect and stop fraud in real-time? This. .. Tagged with beginners, kafka, datascience, dataengineering....

4+ day, 3+ hour ago  (311+ words) Kafka compression waste is usually a batch depth problem, not a codec problem. Better. .. Tagged with dataengineering, distributedsystems, infrastructure, performance....

5+ day, 14+ hour ago  (13+ words) I built a Python decorator that watches your Data Frame pipelines automatically You know. .....

6+ day, 19+ hour ago  (1298+ words) The first time you save a Streamlit file in your editor and watch the browser update before your hand leaves the keyboard, you assume it is some clever diffing magic. It is not. The mechanism underneath is closer to a…...

1+ week, 11+ hour ago  (563+ words) In this post, you learn how Smartsheet built a Real-time Dynamic Filtering (RDF) system on Amazon Managed Service for Apache Flink, cutting messaging costs by over $40, 000 per month and improving live collaboration latency by 1. 8x. Although SNS supports programmatic changes to…...

1+ week, 18+ hour ago  (189+ words) Most standard data pipelines run on a schedule. You use tools like Airflow and dbt to extract and. .. Tagged with eventdriven, kafka, ai, dataengineering....

1+ week, 1+ day ago  (17+ words) Introduction In modern data engineering, handling high-velocity, real-time streams. .. Tagged with luxdev, dataengineering, kafka, docker....

1+ week, 2+ day ago  (882+ words) Building real-time systems is often presented as a distributed systems problem. Kafka, stream processors, event buses, fanout pipelines, multiple caches " the architecture diagrams usually become complicated very quickly. But the problem we were trying to solve was actually much simpler....