Streaming · Real-time

Flight Pipeline

Kafka Streaming · Southwest Airlines · Jan 2023 - Jan 2024

96% Match20231 yr 1 moMTTR 45 to 12 min

Southwest's flight-ops team needed real-time visibility into the Elastic Flat Files (EFFs) that ground systems emit during every flight. We replaced the batch path with a Kafka-based stream that surfaces phase, position, and anomalies within seconds.

Kafka StreamsRabbitMQS3 SinkJava · GradleMTTR 45 to 12 min

Project Overview

Southwest's flight-ops team needed real-time visibility into the Elastic Flat Files (EFFs) that ground systems emit during every flight - currently arriving by FTP, batched, and analyzed hours later. We replaced the batch path with a Kafka-based stream that surfaces phase, position, and anomalies within seconds.

Kafka StreamsRabbitMQ MicroservicesS3 SinkPython · FlaskJava · GradleGitLab CI95% test coverageLocal-stack Kubernetes

Project Preview

WN1234 · DAL-LAXcruise

WN0876 · MDW-DENclimb

WN2103 · BWI-FLLdescent

WN0521 · HOU-PHXcruise

WN1890 · OAK-SANtaxi

Project Details

ClientSouthwest Airlines

RoleSenior SWE / Data Scientist

Team Size12 engineers

Duration1 yr 1 mo

StackKafka · Java · Python

73%

MTTR reduction

<1s

end-to-end latency

Problem Statement

Hours-late insights. EFFs landed by FTP and analytics jobs ran every 4 hours - by the time an anomaly was visible, the flight had landed.
Brittle integration. Existing pipeline was a chain of cron jobs and shared filesystems. A single missed file silently dropped the next 6 hours of data.
No local dev story. Engineers could only test against a shared QA environment, blocking each other constantly.

<1s

End-to-end latency from EFF arrival to dashboard visualization, versus 4-hour batches before.

95%

Test coverage on Java conversion

12K/min

Events sustained

The system, in one diagram.

Four layers: EFF & ACARS ground telemetry ingestion, distributed Kafka event stream buffer, Java/Spring Kafka Streams anomaly detector, and AWS S3 Lakehouse persistence.

Southwest Airlines Real-Time Flight Pipeline Architecture: Ingestion, Kafka Stream Buffer, Java Stream Processing, and S3 Sink

My Role

Senior Software Engineer / Data Scientist on a team of 12. Designed the Kafka topology, owned the EFF → S3 → Kafka pipeline, and led the Python → Java port for the production hot-path.

Kafka TopologyRabbitMQ pub/subStream TransformsLocal-stack DockerTest StrategyData Governance

The Approach

// STEP 01

Pub/sub instead of cron.

RabbitMQ between ingestion services for fan-out (analytics + cold-storage + alerting consumers). Kafka for the high-throughput tracking events. No service knows about another's existence.

// STEP 02

Local S3 in a Docker pod.

Built a local-stack S3 + Kafka inside a multi-pod Kubernetes setup that mirrors prod. Every engineer gets their own fully-isolated environment in under 30 seconds.

// STEP 03

Python prototypes, Java for hot paths.

Prototyped transforms in Python (fast iteration), then ported the proven ones to Java with Gradle builds and 95% unit-test coverage. Best of both speeds.

The Outcomes

<1s

Stream latency

From 4-hour batch to sub-second visibility on every flight event.

95%

Test coverage

Java ports landed with unit suites that catch ~95% of regressions before merge.

12 → 1

Cron jobs replaced

An entire constellation of cron + shared FS jobs collapsed into one event-driven topology.