Data Lake can be implemented in AWS using S3, which is a highly scalable, durable, and secure object storage service. Data is stored in S3 as objects, which can be any type of data, such as text, images, videos, or binary data.
Data lakes can store data in any format, including structured data from relational databases, unstructured data from log files and social media feeds, and semi-structured data from JSON documents.
Lake Formation is a fully-managed service by AWS that provides businesses with a simple and secure way to build, secure, and manage their Data Lakes. It automates various tasks associated with building Data Lakes, such as data ingestion, cataloging, cleaning, and securing.
Lake Formation provides a centralized data catalog that allows businesses to create, maintain, and search for their data assets. It also includes fine-grained access control that helps businesses ensure their data is accessed only by authorized users and that it meets compliance requirements.
AWS Glue Crawler is a serverless data processing service that allows us to automatically discover the schema and structure of our data stored in our data lake. Glue Crawler traverses through data sources and catalogs the metadata of data assets, creating a table that contains the schema information of the data.
Using Glue Crawler, we can create and maintain a data catalog that can be used by other services in our AWS environment, such as Athena.
Athena is a serverless query engine service that enables us to analyze data stored in our data lake using standard SQL queries. Athena is designed to handle large-scale data and can scale automatically to handle large data volumes and high concurrency.
Using Athena, we can analyze our data directly from our data lake without the need to move data into a separate analytics platform or data warehouse.
Steps: Set up Lake Formation and S3 bucket for the data lake, manually upload a sample customer CSV file to the S3 bucket, using Glue Crawler to automatically create a table that contains the schema information of the data, and lastly, using Athena to query and analyze the data.
