Multistage Adaptive Load Balancing in Big Active Data Publish Subscribe Systems

Speaker: Hang
Time: 12:30pm
Room: 3011

We address issues in the design and operation of a Big Active Data Publish Subscribe (BAD Pub/Sub) systems to enable the next generation of enriched notification systems that can scale to societal levels. The proposed BAD Pub/Sub systems aim to ingest massive amounts of data from heterogeneous publishers and sources and deliver customized, enriched notifications to end users that express interests in these data items via parameterized channels. To support scalability, we employ a hierarchical architecture that combines a back-end big data cluster (to receive publications and data feeds, store data and process subscriptions) with a client-facing distributed broker network that manages user subscriptions and scales the delivery process. The skewed distribution of subscribers, their interests and the dynamic nature of societal scale publications, create load imbalance in the distributed broker network. We propose a staged approach for broker load balancing that executes in multiple stages — initial placement of brokers to subscribers, dynamic subscriber migration during operation to handle transient and instantaneous loads and occasional shuffles to re-stabilize the system.