Spike - Integrate Lakekeeper for Catalog management

[irfanghat]

Title: Integrate Lakekeeper for Data Governance in Embra Connect

Description:
We need to integrate Lakekeeper as an alternative to Unity Catalog for data governance in Embra Connect. This will help enforce access control, manage metadata, and ensure compliance within our data infrastructure.

Tasks:

1. Research & Evaluation

   • Assess Lakekeeper’s compatibility with our existing data architecture.
   • Compare features with Unity Catalog.
   • Identify potential integration challenges.

2. Setup & Configuration

   • Deploy Lakekeeper and configure it within the Embra Connect environment.
   • Integrate authentication with OpenID or other authentication mechanisms.
   • Set up role-based access control using OpenFGA or a suitable alternative.

3. Data Governance Implementation

   • Define and enforce data access policies.
   • Configure metadata management and auditing features.
   • Ensure proper permissions are applied to different user roles.

4. Testing & Validation

   • Conduct tests to verify security and compliance.
   • Ensure query engines can access data seamlessly via Lakekeeper.
   • Monitor performance and troubleshoot any integration issues.

5. Documentation & Training

   • Document integration steps, configurations, and best practices.
   • Train relevant teams on managing and using Lakekeeper for governance.

6. Deployment & Monitoring

   • Roll out Lakekeeper in a staged manner.
   • Monitor logs and system performance post-deployment.
   • Gather feedback and iterate on governance policies as needed.

Priority: High
Owner: @irfanghat
Status: Open