Google Cloud Storage target for luigi.
Mostly copy and paste of the S3Target with AWS api calls replaced with GCS calls. It has only two major differences:
- AFAIK we can't make resumable downloads in GCS
for uploads file extention is required to make it possible to guess file mime typeWithout extension file will be treated as binary file
target = GcsTarget(path)
will create target with GCS account details loaded from default config file (/etc/gaw/gapi.yaml
)
fs = GcsFileSystem(conf=dict_conf)
target = GcsTarget(path, gcs_filesystem=conf)
will create target with gcs conf provided by dictionary dict_conf
fs = GcsFileSystem(
secret_key=gcs_secret_key_file_path,
email=gcs_email,
private_key_password=optional_password)
target = GcsTarget(path, gcs_filesystem=conf)
will create target that uses GCS account gcs_email
identified by secret key stored in a file located
at gcs_secret_key_file_path
with password optional_password
(which is "notasecret" by default)
Here is sample YAML file with all required and optional configuration options
gapi:
auth:
cert: './conf/privatekey.pem'
password: 'notasecret'
email: 'app-id@developer.gserviceaccount.com'