Trilogix Cloud

Standard Technology Stack for an AWS based Data Lake

A typical Technology Stack for a Data Lake.  S3 as the Golden Source.  Snowflake as a corporate Data Share with SQL use cases.  If AWS-S3 and Redshift are not properly cross-account, cross region accessible, the data gravity will move to Snowflake as the Golden source with its inbuilt data sharing.  Entitlements or Access controls needs to be designed up front.  Data ingestion tooling should use cloud native or open source – don’t custom build your own.  Data transformation could include dbt, Glue and Databricks.  Analytics can be handled by Databricks especially for semi-unstructure data formats (json, xml).  Consumption patterns need to be understood in depth, including tooling and where the tools are located.  In general, many products and complexity.

 

 

 

Leave a Comment

Your email address will not be published. Required fields are marked *