Bucket Storage similar with S3 at Linode

This shows how to access the bucket storage at Linode with the code you normally use for AWS.

More and more cloud providers are rolling out bucket storage solutions similar with S3. The similarity goes to the point that you can merely repoint your application from AWS related urls to the new ones and it works out of the box.

The only thing needed is to change the configuration. The main points around the new configuration

New endpoint. AWS S3 commands typically work without endpoint, if you selected the correct region. Any provider other than AWS must have the endpoint set.
AWS key and secret key are of course set on the provider. Linode allows setting those keys specifically for the bucket storage service.

The endpoint for the region Newark / New Jersey in Linode looks like that:

https://us-east-1.linodeobjects.com

Note that the region name is us-east-1 in our situation. With this new region, provided that we ran AWS configure and set the right key and secret key (yes, AWS cli works as well out of the box with the Linode bucket services) then this would be the required command to list for example the existing buckets under our account

aws s3 ls --endpoint=https://us-east-1.linodeobjects.com

This lists the buckets inside our linode bucket storage.

Example of accessing bucket Linode services from Java

Here is the maven dependency needed to get this

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.12.470</version>
        </dependency>

And here is the code, including the imports

package org.example;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;

import java.util.List;

public class Main {
    public static void main(String[] args) {
        try {
            new Main().start();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private void start() throws Exception {
        String endpoint = "https://us-east-1.linodeobjects.com";
        String region = "us-east-1";
        BasicAWSCredentials credentials = new BasicAWSCredentials("akey", "secret key");
        AWSStaticCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(credentials);

        AwsClientBuilder.EndpointConfiguration configuration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
        AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard().withCredentials(credentialsProvider);
        AmazonS3 client = builder.withEndpointConfiguration(configuration).build();

        List<Bucket> buckets = client.listBuckets();
        buckets.forEach(bucket -> System.out.println(bucket.getName()));
    }
}

Example of accessing bucket Linode services from Python

There is a package called boto3 which is advertised on AWS website, so this is the standard, it seems. The calls are very easy to instrument, here is a sample

import boto3


def starting():
    client = boto3.client('s3', region_name='us-east-1',
                 endpoint_url='https://us-east-1.linodeobjects.com',
                 aws_access_key_id='key id',
                 aws_secret_access_key='secret access key')

    buckets = client.list_buckets()
    raw_data = buckets['Buckets'];
    for dt in raw_data:
        print('current name: ', dt['Name'])


if __name__ == '__main__':
    starting()

Example of accessing bucket Linode services from Golang

In golang there are a number of libraries recommended to be used to connect to AWS. All of them of course are developed at Amazon. S3, as expected, works everywhere. Here is the program that lists the buckets in a S3 account end to end – including the imports.

package main

import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
err := proceed()
if err != nil {
fmt.Println("there was an error: ", err.Error())
}
}

func proceed() error {
creds := credentials.NewCredentials(&credentials.StaticProvider{Value: credentials.Value{
AccessKeyID:     "access key",
SecretAccessKey: "secret access key",
}})

sess, err := session.NewSession(&aws.Config{
Credentials: creds,
Endpoint:    aws.String("https://us-east-1.linodeobjects.com"),
Region:      aws.String("us-east-1"),
})

if err != nil {
return err
}

svc := s3.New(sess)
out, err := svc.ListBuckets(nil)
if err != nil {
return err
}

buckets := out.Buckets
for _, bucket := range buckets {
fmt.Println(*bucket.Name)
}

return nil
}

Note a couple of things

Endpoint must be mentioned if you do not connect to AWS but to another provider
It is possible to omit the credentials in which case will take them from .aws folder, however in a containerized Docker or Kubernetes solution this is not at all recommended, typically the credentials come from an adjacent artifact like Cyber Ark or similar

The above is just an idea of how to connect. Of course there are a gazillion of methods and packages that support the S3 access in Go but all of them go through this connectivity step.