ShareChat — Kafka + Flink Event Joining at 180M MAUs
ShareChat replaced a Golang/Memcache consumer with Flink for 180M MAU event joining, cutting costs 52% and reducing throughput 85% by switching from JSON to Avro + LZ4.
Scale
180M+ MAU across 18 languages; views stream at 100MB/sec peak
Before
Golang consumer with Memcache — high cost, scaling issues joining views and engagement streams
After
Flink eliminated Memcache entirely; 52% total pipeline cost reduction; switching JSON → Avro + LZ4 reduced throughput 85%
Key Insight
Serialization format and compression choices can be transformational — switching from JSON to Avro + LZ4 reduced throughput 85%. Infrastructure economics are often won at the serialization layer.
In a Snowflake Conversation
This is the economics story — streaming architectures have significant infrastructure costs, and serialization format + compression choices can be transformational. Source: CIKM 2024 workshop paper.
My Read
Practitioner commentary coming soon.
Relevant Conversations