There are a ton of reasons why you should be using object storage right now, faster data retrieval, reduction in cost, optimization of resources, are just a few of those reasons. Due to the scale-out nature of object storage, it's less costly to store all of your data with this technology. Amazon S3 is without a doubt the most know service of this kind out there, however there are other options outside besides from Amazon. The object storage service of providers such as Digital Ocean, Linode and Vultr are just a few of the most trustworthy solutions for object storage.
Manipulating buckets in Amazon is quite easy using the AWS CLI, however, when we talk about the manipulation of buckets in the object storage of other providers, AWS CLI is not an option, instead, all of them rely on the s3cmd command line tool. As it seems to be unclear for a lot of people how to use this CLI tool in Windows, I will explain to you how to easily install and use s3cmd in Windows 10.
1. Download s3cmd
S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects. It is best suited for power users who are familiar with command line programs. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.
You can download the latest available version of S3cmd from the official repository at Github here. Go to the releases page here, I downloaded the
s3cmd-2.2.0.zip file. Then, extract its content in the following directory:
Of course you can change this directory and place it wherever you want, but I'll use this directory in this article. The binary can be used like this:
But it will work only if Python is already installed, if it's not, be sure to follow the next step.
2. Install Python 3
As mentioned in the first step, the latest version of s3cmd 2.2.0 requires Python 3 to work properly. You need to install Python 3 in your system, in case you don't have Python 3 installed already, be sure to download the installer from the official Python website here. For Windows, there's already an user friendly installer that registers automatically the python binary to the PATH, so after the installation, you will be able to use python from the command prompt:
In my case, I installed Python 3.9.8.
Important: you need to install Python 3, otherwise if you run the package using Python 2, you may find the following exception in your terminal:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ImportError trying to import dateutil.parser. Please install the python dateutil module: $ sudo apt-get install python-dateutil or $ sudo yum install python-dateutil or $ pip install python-dateutil !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
3. Proceed with the configuration
In my case, I'm configuring a bucket in the object storage service of Vultr, so the information that I'm going to fill in this step are the required by this service, however it should be similar for the service you decide to use. The binary of s3cmd is the s3cmd file located in the directory where you extracted the downloaded content, just run the script with Python and use the
--configure argument to start configuring the Object Storage credentials:
python C:\s3cmd\s3cmd --configure
Once you finish the questionnaire, you will get an output similar to the following one (in the last step, the connection will be tested):
ERROR: Option --preserve is not yet supported on MS Windows platform. Assuming --no-preserve. ERROR: Option --progress is not yet supported on MS Windows platform. Assuming --no-progress. Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: XXXXXXXXXXXX Secret Key: XXXXXXXXXXXXX Default Region [US]: Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: ewr1.vultrobjects.com Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.ewr1.vultrobjects.com Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [C:\Program Files (x86)\Gpg4win\..\GnuPG\bin\gpg.EXE]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: Yes On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: New settings: Access Key: XXXXXXXXXXXX Secret Key: XXXXXXXXXXXX Default Region: US S3 Endpoint: ewr1.vultrobjects.com DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.ewr1.vultrobjects.com Encryption password: Path to GPG program: C:\Program Files (x86)\Gpg4win\..\GnuPG\bin\gpg.EXE Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Not configured. Never mind. Save settings? [y/N] y Configuration saved to 'C:\Users\sdkca\AppData\Roaming\s3cmd.ini'
4. Try to access your object storage
As final step, try executing a simple operation, like listing the buckets that you have created in your object storage:
python C:\s3cmd\s3cmd ls
In my case, I have a single bucket:
And it works perfectly in Windows!
Happy coding ❤️!