Big data analytics
Overview
The client had set up ‘data’ as a strategic objective to address a number of business issues which could not be resolved with existing systems and reporting processes The client wanted a scalable and future proof platform which could be scaled as needed in a cost-effective manner
What we did
We designed a high performance, highly scalable architecture, designed of optimal cost to operate (c.US$1000 per month) Our DevOps team supported the implementation phase prior to being handed over to the client’s team
Outcome and value
The new platform is allowing the client to explore new business problems (e.g. recovery of customer debt) which were not possible while showcasing the value of more effective delivery methods and highly skilled resources
Tech overview
Automated services load data to the Data Lake in Amazon S3 Amazon EMR with Jupyter notebooks and Amazon Glue process and transform the data Amazon Athena enables users interactive, SQL style analysis of Amazon S3 data Amazon QuickSight allows dynamic, drilldown, pivot table reporting and automated pdf/html reports to stake holders AWS Snowball was used for the initial 20TB data injection into the Data Lake 2TB of monthly data added from multiple vendor database sources with Database Migration Service and csv files via Storage Gateway: File Gateway
Finance analytics
Overview
The finance function was creating a cloud-based analytics platform to provide better insight into their gross margin and enable better decision making in a very completive market This required using 200 data sources, 20Tb of data and very complex calculation. It required a high-performance architecture which could be used with user-friendly tools to develop the analytics models Initial delivery was attempted with existing internal expertise and the main offshore provider but was not progressing at the cost, quality and speed the client required
What we did
Our engagement lead and senior architect took over the design, creating a much more simplified but effective architecture which also prioritised cost optimisation Our DevOps team took over development from the offshore provider and completely transformed the agility and delivery effectiveness We created an environment where the internal team was up-skilled
Outcome and value
Fail fast and fail cheaply was not possible, the offshore provider would take c.6 weeks and c£20k, min for any iteration. We delivered POCs in a day, for less than £2000 each We started the discussion around DevOps in the organisation, which is now trying to adopt it more widely We saved over £130k per year just by designing a more effective data upload process The project was delivered at a fraction of a cost as it would have been done by the offshore provider and the agility we provided allowed to build 3 generations of the architecture for the price of one
Tech Overview
Snowflake data warehouse with 26Tb of data Strong data warehouse roles and security model Optimised cost warehouse utilisation for near zero cost waste Alteryx for analytics models (business requirement: No code). AWS based Alteryx server for orchestration Tableau and Quicksight for reporting and visualisation
