This case study describes an incremental cross‑cloud ingestion pipeline that moves datasets from GCP BigQuery into an AWS data lake, with Apache Iceberg providing table format, snapshots, and schema evolution. AWS Glue (Spark) reads from BigQuery using the official connector, writes partitioned Parquet files to S3, and commits transactions to Iceberg for consistent, query‑ready tables.
Notes
- Apache Iceberg provides snapshot‑based incremental reads, atomic commits, and schema evolution.
- AWS Glue (Spark) jobs are partition‑aware and retry‑safe; reruns are idempotent.
- Observability covers lag, freshness, error rates, and data egress; alerts surface late partitions and schema changes.
