90 mln rows of data
Custom connector development
Multiple data quality checks
DevOps for faster releases
The client is a US-based insurance provider with over 5,000 employees and offices around the world. Founded about a quarter of a century ago, the insurance giant has grown a massive client base and accumulated vast amounts of data.
This strategically valuable data, however, was locked in the client’s Oracle ERP system. The existing setup allowed only for very limited customization and required very labor-intensive data processing workflows. And even so, many attributes and metrics important for further decision-making and refining the value chain remained unavailable.
To capitalize on this data treasure trove and open the door to new insights, the client needed to set up an effective and highly-performant pipeline to extract comprehensive ERP data, clean it and load it into the Financial Analytics Data Warehouse for further analysis. The data-heavy project called for mature ETL development expertise, one of Symfa’s core competence.
Although the fundamental principles of any ETL project are relatively simple, the challenge rises with the complexity and the amount of data. This was exactly the case for our client as being a global carrier, the client’s sources generate a huge amount of complex financial data.
The Symfa team embarked on the project to streamline the data flow between Oracle ERP and the client's custom data warehouse sitting on the SQL server. Our scope of work included:
To ensure stable and reliable connection with the database, our team built a complex custom connector from the ground up. The connector works like clockwork, automatically restarts, if necessary, and verifies the data in case of any errors.
The ETL pipeline built by the team seamlessly extracts data from Oracle ERP, cleans it, maps against the target model and loads the data into the Financial Data Analytics Warehouse.
To guarantee data accuracy and completeness, the ETL solution includes multiple data quality checks. In addition, emails are automatically sent after every ETL process with the links to data quality and performance reports.
To accelerate delivery and increase efficiency, the team set up a reliable CI/CD pipeline that leveraged automated release management, including automated build and deployment, granular permissions and access levels, and more.
With data being at the core of the modern insurance business, ETL projects recently rose to prominence in the client’s company, being the cornerstone of data-driven culture. Now, this robust and reliable ETL pipeline successfully fuels the client’s far-reaching analytical plans. With the ability to transform diverse financial data into analytics-ready data that can be acted upon, the client can inform its strategic planning and open up new revenue streams.
From the perspective of collaboration with the client’s specialists responsible for further analysis, our engineers have become an integral part of the distributed team, enabling an organic data flow and providing each other with extra support whenever it is needed.
Our team will get back to you promptly to discuss the next steps