Our solution was to bring everything together in one place.
We created data pipelines to supply an Integrated Consumption Data Store (ICDS) that supported GFS's operational and analytical needs. In essence, everything collected flowed into one place where it could be viewed, used and analyzed to add real value for decision-makers.
The tech that powered the data modernization
Understanding the technology available and working closely with those who create it means the optimum solution can be delivered. To build the ICDS, we leveraged our long-term partnership with Google Cloud.
After careful analysis and consultation, we agreed that Google Cloud’s Dataflow would be ideal as a serverless execution engine for Apache Beam SDK to do batch data processing.
Data is extracted from BigQuery and stored in Cloud SQL. This allows GFS to achieve data syndication, speed and accessibility for various operational needs.
Faster, flexible and cost-effective
Not only does this solution save GFS money, but it can be scaled up and down when required. Data is accessed via a standardized API layer built and deployed within GKE. This takes advantage of the automated scaling and high-availability across regions and multiple zones.
Costs and resource overhead are managed on database admins, and it delivers data significantly faster while using BigQuery for limitless compute along with Dataflow for ETL processing of batch and streaming data. CI/CD automation is integrated into the data pipelines.