Cloud storage refers to storing your data in remote locations i.e., the cloud. Cloud Storage became increasingly popular in the information era. With tons of data in hand, storing them in the cloud is very affordable. By storing the data in the cloud, we can access the data at any time irrespective of the device and the location.
Google Cloud Storage (GCS)
Google Cloud Storage is a service provided by the Google Cloud Platform(GCP) for cloud storage. GCP offers pay-per-use services. Apart from Cloud Storage, GCP offers many other services like hosting your websites in the cloud, running your own Operation System from the Cloud, and other cloud computing processes.
Objects in GCS
An object is any kind of data that you upload to Google Cloud Storage. For example, an object can be a file, image, video, application, etc. Every object has data, metadata, a unique identifier, etc. For more details like naming conventions etc., you can refer to your cloud storage service provider. (Here it is GCP).
Pre-Requisites
- A Google Cloud account with billing enabled
- Sample object to upload and download from GCS. (Here I am using a sample.txt file as the object)
- A basic idea on the command line and API.
- Have created your bucket in GCP.
Uploading Objects in Google Cloud Storage
Command Line
GCP provides Google Cloud CLI to interact with Google Cloud Storage through the command line. Google Cloud CLI includes the gcloud, gsutil and bq command-line tools. Here, I will use gsutil command tool to upload the object. For illustration, I am using sample.txt file. The file contains ‘This is a sample text’ as data.
Step 1: Use the below command to upload the object to gcs:
gsutil cp [LOCAL_FILE] gs://[BUCKET_NAME]/The above command will upload LOCAL_FILE to the bucket named BUCKET_NAME.
Example: I have created a bucket named gfgbucket and uploaded sample.txt as below
gsutil cp [sample.txt] gs://[gfgbucket]/Output:
Copying file://sample.txt [Content-Type=text/plain]...
/ [1 files][ 22.0 B/ 22.0 B]
Operation completed over 1 objects/22.0 B.
Goto the bucket from the cloud console to see our sample.txt as below
Object upload
API
You can access Google Cloud either through client libraries or REST APIs. Here I will illustrate how to use both the methods.
Client Libraries:
Client libraries are inbuilt libraries in the programming languages to access the Google Cloud Storage. Currently, GCP provides client libraries in C++, C#, Go, Java, Node.js, PHP, Python and Ruby. Below is the sample code in Java and Python using Google Client Library.
Step 2: Write the code in any of the following languages
import com.google.cloud.storage.Blob;
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import java.nio.file.Path;
import java.nio.file.Paths;
public class GCSUploader {
public static void main(String[] args) {
// Replace these values with your actual GCP project ID and bucket name
String projectId = "your-project-id";
String bucketName = "your-bucket-name";
// Path to the local file you want to upload
String localFilePath = "path/to/local/file.txt";
// Destination object name in the Cloud Storage bucket
String objectName = "your-object-name";
// Create a Storage client
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
// Get a reference to the target bucket
Bucket bucket = storage.get(bucketName);
// Get a Blob reference and upload the file
Blob blob = bucket.create(objectName, Paths.get(localFilePath));
System.out.println("File uploaded to Cloud Storage: " + blob.getName());
}
} |
# Importing the Google Cloud client library from google.cloud import storage
# Initializing the client client = storage.Client()
# Getting the bucket bucket = client.get_bucket('[BUCKET_NAME]')
# Creating a blob (object) in the bucket blob = bucket.blob('[OBJECT_NAME]')
local_file_path = 'path/to/local/file.txt'
# Uploading the local file to the blob blob.upload_from_filename(local_file_path) print(f'File uploaded to Cloud Storage: {object_name}')
|
Output:
File uploaded to Cloud Storage: sample.txt
REST API:
Through REST API, you can directly make HTTP request to Google Cloud Storage.
Step 3: Use the following cURL command to make HTTP request:
curl -X POST --data-binary @[LOCAL_FILE] -H
"Authorization: Bearer [YOUR_ACCESS_TOKEN]" -H
"Content-Type: [CONTENT_TYPE]" "https://storage.googleapis.com/upload/storage/v1/b/[BUCKET_NAME]/o?uploadType=media&name=[OBJECT_NAME]
In the above command, LOCAL_FILE is the path to the local file you want to upload, YOUR_ACCESS_TOKEN is authorization token that you get from your cloud service provider, OBJECT_NAME is the name you want to give to the object in your Google Cloud Storage Bucket.
Example: Let us upload sample.txt as sample2 to the bucket as below
curl -X POST --data-binary @sample.txt \
-H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" \
"https://www.googleapis.com/upload/storage/v1/b/gfgbucket/o?uploadType=media&name=sample2"
Output:
{
"kind": "storage#object",
"id": "gfgbucket/sample2/1702468444308874",
"selfLink": "https://www.googleapis.com/storage/v1/b/gfgbucket/o/sample2",
"mediaLink": "https://www.googleapis.com/download/storage/v1/b/gfgbucket/o/sample2?generation=1702468444308874&alt=media",
"name": "sample2",
"bucket": "gfgbucket",
"generation": "1702468444308874",
"metageneration": "1",
"contentType": "text/plain",
"storageClass": "STANDARD",
"size": "22",
"md5Hash": "kzOPKF76QRKlpptuJ0TT0w==",
"crc32c": "ru17ew==",
"etag": "CIqL9ZatjIMDEAE=",
"timeCreated": "2023-12-13T11:54:04.352Z",
"updated": "2023-12-13T11:54:04.352Z",
"timeStorageClassUpdated": "2023-12-13T11:54:04.352Z"
}
Downloading Objects from Google Cloud Storage
Downloading objects from Google Cloud Storage is similar to uploading. Below is the sample illustration.
Command Line
Step 1: Use the below command to download the object from gcs:
gsutil cp gs://[BUCKET_NAME]/[OBJECT_NAME] [LOCAL_DESTINATION_PATH]
The above command will download the object BUCKET_NAME/OBJECT_NAME to your LOCAL_DESTINATION_PATH.
Example: Let us download sample.txt to a sampleFolder from the bucket as below
gsutil cp gs://gfgbucket/sample.txt ./sampleFolder
Output
Copying gs://gfgbucket/sample.txt...
/ [1 files][ 22.0 B/ 22.0 B]
Operation completed over 1 objects/22.0 B.
API
Client Libraries:
Step 2: Use the below sample code written in Java and Python using Google Client Library to download your object.
import com.google.cloud.storage.Blob;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import java.nio.file.Path;
import java.nio.file.Paths;
public class GCSDownloader {
public static void main(String[] args) {
// Replace these values with your actual GCP project ID and bucket name
String projectId = "your-project-id";
String bucketName = "your-bucket-name";
// Name of the object in the Cloud Storage bucket that you want to download
String objectName = "your-object-name";
// Path to the local file where you want to save the downloaded file
String localFilePath = "path/to/local/downloaded-file.txt";
// Create a Storage client
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
// Get the Blob reference and download the file
Blob blob = storage.get(bucketName, objectName);
blob.downloadTo(Paths.get(localFilePath));
System.out.println("File downloaded from Cloud Storage: " + blob.getName());
}
} |
# Importing the Google Cloud client library from google.cloud import storage
# Initializing the client client = storage.Client()
# Getting the bucket bucket = client.get_bucket('[BUCKET_NAME]')
# Getting the blob (object) in the bucket blob = bucket.blob('[OBJECT_NAME]')
# Downloading the blob to a local file blob.download_to_filename('[LOCAL_DESTINATION_PATH]')
print(f'File downloaded from Cloud Storage: {blob.name}')
|
Output:
File downloaded from Cloud Storage: sample.txt
REST API:
Step 3: Use below command to download object using cURL command.
curl -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]"
"https://storage.googleapis.com/storage/v1/b/[BUCKET_NAME]/o/[OBJECT_NAME]?alt=media" -o [LOCAL_DESTINATION_PATH]
Example: Let us download sample.txt to our sampleFolder as sample2 from the bucket as below
curl -H "Authorization: Bearer [YOUR_ACCESS_TOKEN]" "https://storage.googleapis.com/storage/v1/b/gfgbucket/o/sample2?alt=media" -o ./sampleFolder
Output:
ESFQHGX2Mir3gF5erD8xeoc8M1QolA1w0171" "https://storage.googleapis.com/storage/v1/b/gfgbucket/o/sample2?alt=media" -o ./sampleFolder/sample2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22 100 22 0 0 87 0 --:--:-- --:--:-- --:--:-- 87
FAQs on Uploading and Downloading Objects in Google Cloud Storage: Command Line and API
1. What is Cloud Storage?
Cloud storage refers to storing your data in remote locations i.e., the cloud. The best example is Google Drive. We can store our images, videos and any data in Google Drive.
2. What is Object in GCS?
An object is any kind of data that you upload to Google Cloud Storage(GCS). An object can be a file, image, video, application etc.
3. Do I need to enable billing for Cloud Storage?
Yes. Your project should have a billing account to create a bucket and upload or download objects.
4. How to estimate cloud storage charges?
You can visit https://cloud.google.com/storage/pricing. This is Cloud Storage pricing page. Here you can learn how cloud storage calculate price for various services.
5. What is the purpose of Google Cloud Storage?
The purpose of Google Cloud Storage is different from other cloud storages like Google Drive. GCS is primarily meant for storing objects not just images, videos or files. Object storage is useful for scenarios like uploading or downloading data through APIs and programming languages.