A useful architecture to move data from on-premises to AWS is to consider using AWS S3 outputs and move data directly over a Direct Connect to S3 in AWS.
This could be useful to migrate historical data, seed data in the cloud, or move batch processed data.
We can use S3 outposts on premises and connect to an S3 raw bucket in AWS in a landing zone or secured VPC.
eg Batching
Batch processing is commonly used when immediate real-time analytics results are not required
Suitable for scenarios where data can be collected over time and processed in batches, such as aggregating daily sales data or generating reports
Batch processing often involves processing data in large blocks or batches that have been collected over a specific time period
It is a method of analyzing and manipulating significant volumes of data at once, rather than processing it in real-time or as it arrives
There is a variantion in frequency can be daily, weekly or monthly or a combination of all 3
To implement batch processing using AWS, we can use various services such as Amazon EMR, Amazon Redshift, and AWS Glue etc.
Batching and AWS S3 on prem to S3 cloud | Trilogix Cloud
