Why this decision matters
On Google Cloud, moving data in is usually the first job in any project. The exam guide lists "Choose the appropriate data transfer tool (e.g., Storage Transfer Service, Transfer Appliance)" under item 1.1. In practice you are choosing between two things:
- Online transfer: data travels over a network into Cloud Storage. The main managed tool is Storage Transfer Service.
- Offline transfer: data is copied onto a physical device, which is then shipped to Google. That device is Transfer Appliance.
Three inputs drive the choice: how much data you have, how much network bandwidth you can give the transfer, and how much time you have.
Task 1: Estimate how long an online transfer would take
Do this calculation before you pick a tool.
- Find the data volume in terabytes (TB).
- Find the usable bandwidth. This is the share of the link you can actually give the transfer, not the full line speed.
- Use this rule of thumb: 100 Mbps moves about 1 TB per day, and 1 Gbps moves about 10 TB per day.
- Divide volume by daily throughput to get the number of days.
Worked examples:
| Data | Bandwidth | Approx. time online |
|---|---|---|
| 5 TB | 1 Gbps | ~half a day |
| 300 TB | 100 Mbps | ~9 months |
| 1 PB | 100 Mbps | ~3 years |
Google's own documentation uses the same figures: about 9 months for 300 TB at 100 Mbps, and about three years for 1 PB on a typical 100 Mbps network.
Task 2: Apply the decision rule
Work through these checks in order:
- Small, one-off upload from your own machine (up to a few TB): use the gcloud CLI (for example
gcloud storage cp) or the console. You do not need a managed service. - The upload would take more than about one week over the network: choose Transfer Appliance. This is Google's stated threshold.
- The upload finishes in a reasonable time online, especially when the source is another cloud or the copy must repeat on a schedule: choose Storage Transfer Service.
Think of it as postage versus email. A spreadsheet goes by email. A warehouse of filing cabinets goes on a lorry, because the lorry gets there sooner.
Task 3: Know what each tool does
Storage Transfer Service (online, managed)
- Copies data into Cloud Storage from Amazon S3, Microsoft Azure Blob Storage, S3-compatible storage, HDFS, a URL list, on-premises file systems, or other Cloud Storage buckets.
- Runs once or on a schedule, which suits ongoing syncs.
- Is a managed service: it retries failed files and keeps detailed transfer logs.
- Agents: transfers from Amazon S3, Azure, URL lists or Cloud Storage need no agents. Transfers from file systems (on-premises), S3-compatible storage or HDFS need agents and agent pools, which you install on machines that can read the source data.
Transfer Appliance (offline, physical)
- Google ships a high-capacity storage device to your data centre. Models include the TA40 and TA300, in rackable and freestanding versions.
- You copy data onto it over your local network. Linux and macOS use NFS; Windows uses SMB, SCP or SSH.
- The data is encrypted with AES-256, and you manage the keys through Cloud KMS.
- You ship the device back. Google uploads the data to your Cloud Storage bucket, then wipes the device to the NIST 800-88 standard.
- Best for very large datasets, slow or expensive links, and data-centre exits.
Task 4: Set up a Storage Transfer Service job (console)
Scenario: copy a nightly export from an Amazon S3 bucket into Cloud Storage.
- Open the Google Cloud console and go to Storage Transfer Service, then click Create transfer job.
- Choose the source type (here, Amazon S3) and the destination type (Google Cloud Storage).
- Configure the source: enter the bucket name and give access credentials. For an on-premises file system source, pick an agent pool and the source directory instead.
- Choose the destination: select the Cloud Storage bucket, and optionally a folder.
- Choose the schedule: Run once, Run every day, Run every week, or Run with custom frequency. Then set it to start now or on a chosen date.
- Choose the settings:
- Overwrite: If different (only replace files whose checksums differ) or Always.
- Delete: Never, Delete files from source after they're transferred, or Delete files from destination if they're not also at source (this keeps the destination an exact mirror).
- Logging and Pub/Sub notifications, if you need them.
- Click Create. Then check progress in the job's run history.
Safety tip: the two delete options remove data. Leave Never selected unless the requirements clearly ask for a move or a mirror.
What the exam asks about this
Expect short scenarios that give a data size, a bandwidth and a deadline. To answer them:
- Do the rough arithmetic. If the online transfer takes more than about a week, or would overrun the deadline, choose Transfer Appliance.
- If the data is in another cloud (S3, Azure) or the copy must recur, choose Storage Transfer Service.
- Watch for these distractors:
- BigQuery Data Transfer Service loads data into BigQuery, not bulk files into Cloud Storage.
- Database Migration Service migrates databases.
- gcloud is fine for small one-off copies but is not a managed, scheduled, large-scale option.
- "Buy more bandwidth" is rarely the intended answer when a deadline is tight.
Key takeaways
- Online transfer uses Storage Transfer Service; offline transfer uses Transfer Appliance.
- Rule of thumb: 100 Mbps moves about 1 TB per day.
- More than about one week to upload means Google recommends Transfer Appliance.
- Storage Transfer Service needs agents for file-system, S3-compatible and HDFS sources, but not for S3, Azure, URL lists or Cloud Storage.
- Job setup order: source, then destination, then schedule, then settings, then create.
- Treat the delete options with care: the default Never is the safe choice.