- Financial Services
Reducing data processing time from 10+ hours to minutes for a pricing consultancy
Overview
Unlocking faster, scalable pricing insights
A strategic pricing consultancy faced challenges in processing large-scale customer pricing data. Their existing solution relied on Python scripts and Excel, which struggled to handle the increased data volume from a new customer. Processing times ballooned to over 10 hours, making iterative analysis impractical and hindering their ability to provide timely pricing insights.
The consultancy needed a flexible, scalable, cloud-based solution that could process data rapidly, support iterative experimentation, and lay the groundwork for a pricing platform that could accommodate additional customers while maintaining data security and isolation.
Our Solution
A modern ELT pipeline to optimize data processing
OneSix designed and implemented a modern ELT (Extract, Load, Transform) data processing pipeline leveraging Azure Data Factory and Snowflake. The architecture streamlined data ingestion, transformation, and analytics, delivering rapid performance and scalability. Key elements of the solution included:
- Automated Data Ingestion: Azure Data Factory ingested raw customer data on a scheduled basis into the client’s Snowflake environment.
- Medallion Architecture: The pipeline followed a medallion architecture (bronze, silver, gold layers) to clean, enhance, and transform raw data into actionable pricing insights and recommendations.
- Data Processing: Processing was conducted with a combination of Snowflake SQL and Python for flexible and powerful transformations.
- Interactive Analytics: A Snowflake Streamlit application allows customers to view insights, experiment with pricing configurations, and analyze results in real time.
- Seamless Data Export: Azure Data Factory exported processed data products into the customer’s SQL database for further analysis and reporting.
- CI/CD Integration: The entire Snowflake environment was managed using CI/CD pipelines via GitHub workflows and Snowflake’s Git integration tools for declarative schema management.
Technologies Implemented
- Snowflake
- Python
- Azure Data Factory
- Streamlit
Results
Achieving fast performance, enhanced agility, and seamless scalability
The solution drastically improved data processing efficiency, reducing execution time from over 10 hours to just minutes or seconds. This rapid performance enabled:
- Faster Iterative Analysis: Clients could quickly experiment with changes to the pricing engine and configurations, facilitating a more iterative and agile approach to pricing strategy.
- Enhanced Confidence: The ability to rapidly analyze data and refine pricing models helped build confidence in the pricing recommendations and insights provided.
- Scalability and Flexibility: The architecture laid the foundation for onboarding additional customers seamlessly, with reusable components and strict data isolation for security.
The consultancy praised the performance and adaptability of the new solution, highlighting how the architecture enabled rapid, iterative development and delivered consistent, high-quality pricing insights.
"OneSix’s solution transformed our data processing capabilities. What used to take hours now takes seconds, allowing us to experiment and deliver pricing insights with unprecedented speed and accuracy."
Ready to unlock the full potential of data and AI?
Book a free consultation to learn how OneSix can help drive meaningful business outcomes.
