Insurance Data Platform for Deep Financial Insights on Demand
Highlights
TURNKEY DELIVERY (INCL. SOLUTION DESIGN)
DATA FOR 5 BUSINESS LINES
TEAM OF 23 (DEV, QA, BA, DM)
5 CUSTOM BUILT ETL PIPELINES
USA
- 1.5 years (ongoing)
Insurance giant embarks on a holistic data structuring project
Our client is an award-winning insurance major operating in the US and European markets. The carrier chose Symfa to carry out a series of renovation projects as a part of their digital transformation initiative, to which this project belongs.
In order to facilitate analytics and reporting for their financial and operational lines, the client embarked on a huge data structuring endeavor. The project covers several insurance business lines (Warranty, PI, Property, M&C, LE, and A&H). The vast amount of data used to come each month to the client’s European offices from authorized agents, all in disparate formats. The project goal was to streamline the data flow and provide the client’s talents with clean data for insightful reports.
Quick system overview and the business value it creates
This is a non-GUI system that processes data behind the scenes, invisible to the business users. It inputs data automatically at a fixed interval (currently once every hour). The interval is chosen based on the users' preference and server's capacities.
The solution for this project involves ETL-based files processing that consists of: a) data extraction from various file sources; b) data transformation according to a specific business model; c) data loading into a database and output files with templates.
Symfa helps the client move the application to the cloud
After evaluating the benefits that the cloud offered for this project, the customer decided to move it partially to the cloud. To do so, the client hired the third-party vendor whom we assisted with the PoC and the cloud infrastructure setup.
The Challenges We Faced Along the Project
With terabytes of data in question and 5 business lines depending on the quality work of the solution, we expected a few challenges along the project road. Mainly, those related to the nature of data and the new insights perspectives that the business wanted to get.
Data intake mechanism
The system required a process that would correctly load extra large files (up to several TB) with data contained in various formats. A by-pass script was created for password-protected files.
Data logic discrepancies
Initially, the team planned to reuse the logic patterns adopted in the client's US offices. The proposed pattern didn't work out for the UK office (the end user of the system). After a series of consultations with the business, the Symfa team went with a proprietary solution design.
Data filling rules
Source files could have empty columns, would come in unsupported languages or simply had the wrong title that the system wouldn't recognize. The Symfa team worked closely with the business units on the data filling rules for the best output data quality and provides data filling consultations whenever the operations ask for new data perspectives.
New ETL pipelines
The project started with one ETL process, yet as the client started to see the real value of the solution, it grew massively. Now we process data in 5 environments representing 5 ETL processes: from dev environment to preproduction to cloud. All the data within those environments are properly synchronized.
Technologies
DevOps effort of the Symfa team
The Symfa team established their own DevOps processes running in Azure DevOps for deployment and control over the current project status. Specific tasks require custom-built solutions, which our engineers occasionally write for this project in C#.
Clear DevOps processes guarantee transparency for everyone on the team and the customer, too.
Project growth through motivated synergetic teamwork
During the first year, the team has implemented all the initial scope of work specified by the client. The Symfa team now is working to make the data intake and processing workflow fully autonomous. They have introduced additional data jobs (now totals 5) to enable a more human-independent system operation. More data jobs are underway to automate repetitive and time-consuming data processing jobs.
To streamline the work of a team of 23 people is no easy job. That’s why we’ve established clear rules for everyone to follow – consistency and clarity in the way we use Scrum boards, tags and the way we communicate in the project chats. Besides that, you really cannot overestimate the value of thorough documentation and a healthy onboarding process for the new-comers.
Latest projects
BACK TO PORTFOLIOContact us
Our team will get back to you promptly to discuss the next steps