How to implement security as a code?

[renepajta]

Permissions are all over the place. We have a data lake with RBACs and ACLs across many folders. we allow to deploy other forms of storage (e.g. SQL DB, MySQL, PosgreSQL, Synapse, etc. ) which have their own access model.

How do we (Data Product team) ensure that the access model is properly moved from DEV to Production environments? Where should be definitions stored?

[marvinbuss]

Thanks for the question @renepajta! In general, we are recommending to push the access definitions down to storage (RBAC and ACLs), use an identity driven access model and use storage as a general medium to share data between producers and consumers.
For more granular access requirements, POSIX ACLs should be used to grant access to the respective table and folder that resides inside the lake.

Certain scenarios may require a compute-driven (SQL DB, MySQL DB, PostgreSQL, MariaDB, Synapse SQL Pool) and not a storage-driven access model in order to add data masking as well as CLS and RLS. In these scenarios, a SQL Pool or other compute may be used to share data between data producer and data consumer. We are still recommending to use an identity driven access model for these services. You can use nested security groups in a similar way to how you would use it on the storage layer to grant access rights to data consumers.

I would like to understand why access permissions need to be moved from a development to a production environment. Why do consumers require access to a dev environment or dataset. For data engineering or data science you require access to real production data. Otherwise, it will become difficult to build useful data science models or optimize your data pipelines.
Please provide more details on why this would be required. Thanks! 👍