Migrating The Buzz NextGen AI Process to AWS
Buzz Indexes
BUZZ Indexes creates and maintains quantitative portfolio strategies based on Big Data analytics models. These models leverage Artificial Intelligence to derive actionable investment insights from alternative datasets including online content and social media. These insights are in turn used as the basis of investment products such as exchange-traded funds.
Brief results of the collaboration:
By migrating an existing machine learning pipeline to the AWS cloud, the company drastically improved the scalability and stability of their machine learning processes.
The delivered solution also enabled Buzz Indexes to “drill down” into collected data which strengthened model evolution and research efforts.
The Need
After performing initial testing and validation of their machine learning pipelines, the Buzz Indexes team deployed their system using on-premise infrastructure. While this environment served the initial needs of the application, the Buzz team was looking to improve two aspects of their existing setup.
Pipeline stability & scalability - Ensure pipeline runs are consistently successful and can scale automatically to handle dynamic workloads and increase in activity.
Data analysis - Being able to understand and drill down into collected data, as well as derive meaningful insights from this data to improve existing models.
The Challenge
Financial models are typically validated over large datasets representing numerous years of financial data which can be extremely compute intensive to process. Additionally, Buzz Indexes required the ability to regularly back-test and iterate over models with agility and speed.
The Solution
To improve pipeline stability & scalability, the Buzz AI process was first deconstructed into multiple components, refactored where required, and converted into EC2 and serverless workloads in the AWS Cloud. Additionally, CI/CD pipelines were developed to enable frequent, low-risk modifications to deployed components.
To enable monitoring and alerts for the entire operation as well as enhance data exploration capabilities, a Grafana dashboard was configured to report on pipeline metrics and collected data, enabling Buzz Indexes to derive key insights from their data.