Performing Bulk Transfers from On-Prem or Other Clouds- completed Flashcards
What is Storage Transfer Service (STS) in GCP?
Storage Transfer Service (STS) is a fully managed solution in GCP that enables efficient, secure movement of large datasets from on-premises systems, other cloud providers like AWS or Azure, or even other GCS buckets into Google Cloud Storage.
What are the key features that STS provides for data migration?
STS offers high-performance bulk transfers, incremental syncs after initial migration, metadata preservation, encryption during transit, integrity validation, event-driven transfers, and built-in monitoring through Cloud Logging and Cloud Monitoring.
Explain how STS ensures data integrity and security during transfers.
STS encrypts all data in transit and runs automatic integrity checks using file checksums after transfer. It also ensures secured agent communication via HTTPS (port 443) for on-prem transfers.
What is incremental transfer in STS? How does it help in large migrations?
Incremental transfer means after the first bulk migration, only newly added or modified files are moved in subsequent runs. It reduces bandwidth usage, speeds up syncs, and keeps source and destination almost real-time synchronized.
How would you set up an on-premises to GCS transfer using STS?
First, I would enable the Storage Transfer API. Then, I’d install Docker and configure STS agents on a Linux machine with access to the local file system. I’d make sure the agent can connect to GCP over HTTPS, configure the source and destination in the STS job settings, and finally create a transfer job via Console or gcloud.
Can you create a sample gcloud command to trigger a transfer job?
Sure.
```bash
gcloud transfer jobs create \
–source-agent-pool=MY_AGENT_POOL \
–source-root-directory=/local/folder/path \
–destination=gs://my-destination-bucket \
–project=my-project-id \
–display-name=”On-Prem-to-GCS-Transfer”
~~~
What ports must be opened on the firewall for the STS agent to work?
Port 443 (HTTPS) must be open to allow the STS agent to securely communicate with Google Cloud.
If a file gets updated after it has been transferred, how does STS handle it?
During scheduled incremental syncs, STS identifies updated files based on metadata and re-transfers only those modified files to GCS.
What happens if the transfer is interrupted mid-way? Does STS handle retries?
Yes, STS has built-in retry mechanisms. It resumes the transfer intelligently without re-uploading already completed files.
You have frequent data updates on your on-premises system. How would you design the transfer to GCS for minimal lag?
I would configure a scheduled transfer job to run at frequent intervals, say every few minutes or hours, depending on the business need, leveraging STS’s incremental transfer capability.
If you need to transfer from AWS S3 to GCS, what specific access setups are required?
I’d set up an access key and secret for the S3 bucket or configure identity federation between AWS and GCP, ensuring proper IAM roles for reading the S3 source.
How would you optimize large-scale data transfers using STS?
I would deploy multiple STS agents for parallelism, use agent pools for balancing workloads, configure chunked uploads, and schedule transfers during off-peak network hours to maximize throughput.
What monitoring mechanisms are available for a Storage Transfer job?
We can monitor transfer jobs in the GCP Console under Storage Transfer, and also integrate with Cloud Logging and Cloud Monitoring for detailed logs, alerts, and dashboards.
How can you schedule and automate recurring transfers using STS?
While creating the transfer job, we can define a schedule for daily, hourly, or even custom CRON-based triggers so that the transfer happens automatically at set intervals.
What IAM roles are required to perform a bulk transfer with STS?
At minimum, roles like Storage Admin or Storage Object Admin on the destination bucket, and permissions like storagetransfer.admin for managing transfer jobs, are needed.
When would you choose STS over tools like gsutil or gcloud storage?
I would prefer STS when handling very large-scale migrations (hundreds of terabytes or petabytes), needing incremental syncs, or when a managed, scheduled, fault-tolerant solution is required.
What are the differences between STS transfers from on-premises vs cloud-to-cloud?
For on-premises, STS requires setting up agents that scan and upload data securely to GCS. For cloud-to-cloud, STS connects directly using credentials, without needing on-prem agents, and pulls data over secure APIs.
If downtime is critical, how does STS help minimize disruptions during migration?
By supporting incremental transfers and scheduling syncs, STS ensures that after the initial heavy migration, only delta changes are synced frequently, thus minimizing downtime and allowing a quick cutover.
