Medtech makeover: Insightec's journey to streamlined data solutions

The implementation of the Calibo platform was designed to build a centralized data lakehouse and insights hub to enhance the overall data management process.

The Calibo platform implementation at Insightec streamlined and automated several key aspects of data management. With its layered architecture, seamless tool integration, and automated workflows, Insightec successfully transformed its data operations to support its growth objectives efficiently. 

This strategic shift enhanced their ability to generate valuable insights while taking full advantage of their diverse data assets.

Implementation  

Building the data architecture: 

• Data sources: The architecture incorporated multiple data sources, including SQL Server, Salesforce, and manual CSV/Excel files. These diverse inputs were vital for bringing together various datasets under one cohesive system.

• Data layering: the data integration workflow was segmented into several key layers: pre-landing, landing, curated, data hub, and serving layer. This structured approach ensured systematic data ingestion, integration, transformation, and quality validation to produce a unified and curated SSOT.

• Snowflake integration: Calibo utilized Snowflake's data platform to orchestrate powerful data transformation algorithms implemented to take advantage of its computational engine to avoid extensive data movement and reduce cost implications.

Using Calibo's platform: 

• Capabilities: The platform allowed for the use of a robust infrastructure with data connectors, data pipeline studios, and DevOps features. This provided a comprehensive framework for managing data workflows efficiently.

• Data pipeline management: Calibo’s Data Pipeline Studio was instrumental in orchestrating workflow streams and implementing dependency management between multiple data pipelines. This ensured seamless data flow and transformations from source to target.

• Tool integration: Tools for source code management (Bitbucket) and CI/CD pipelines were integrated to support deployment activities and maintain version control, further enhancing the agility of the data management process. 

Data quality assurance: 

• Data quality checks: Calibo established frameworks for executing systematic data quality checks – including data quality profiling, analysis, and resolvers. 

• Dashboard for monitoring: A dashboard was implemented to track data quality issues, offering high visibility for the stakeholders to identify and rectify discrepancies proactively. 

Security and access control: 

• Role-Based Access Control (RBAC): The implementation included a robust security protocol with RBAC mechanisms to regulate access across different environments and ensure the integrity of data processing activities.

• Audit logs: Comprehensive audit logs were maintained to provide traceability and oversight of user activities, enhancing data security and compliance. 

Development and deployment automation: 

• CI/CD Processes: Deployment processes were automated using CI/CD pipelines, allowing code to be seamlessly promoted through different environments (from development to production) with minimal manual intervention.

• Pipeline scheduling: Both time-based and event-based scheduling mechanisms were applied to automate data pipeline jobs ensuring reliable and timely data updates. 

Visualization and reporting: 

• Insights Hub: The curated and unified SSOT provided a mechanism for enabling a serving layer that powered the Insights Hub with dashboards creating using Power BI. These dashboards offered an interactive interface for users to visualize data insights, ensuring comprehensive management reporting capabilities.