Skip to content

nkdhny/gcstarget

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 
 
 
 
 

Repository files navigation

GcsTarget

Google Cloud Storage target for luigi.

Mostly copy and paste of the S3Target with AWS api calls replaced with GCS calls. It has only two major differences:

  • AFAIK we can't make resumable downloads in GCS
  • for uploads file extention is required to make it possible to guess file mime type Without extension file will be treated as binary file

How to use

target = GcsTarget(path)

will create target with GCS account details loaded from default config file (/etc/gaw/gapi.yaml)

fs = GcsFileSystem(conf=dict_conf)
target = GcsTarget(path, gcs_filesystem=conf)

will create target with gcs conf provided by dictionary dict_conf

fs = GcsFileSystem(
  secret_key=gcs_secret_key_file_path, 
  email=gcs_email, 
  private_key_password=optional_password)
target = GcsTarget(path, gcs_filesystem=conf)

will create target that uses GCS account gcs_email identified by secret key stored in a file located at gcs_secret_key_file_path with password optional_password (which is "notasecret" by default)

Sample config

Here is sample YAML file with all required and optional configuration options

gapi:
  auth:
    cert: './conf/privatekey.pem'
    password: 'notasecret'
    email: 'app-id@developer.gserviceaccount.com'

About

Google Cloud Storage target for luigi.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages