Use Amazon S3 or Minio<https://minio.io/> as a filestore for resources.
Requires CKAN 2.8+
To install ckanext-s3filestore:
Activate your CKAN virtual environment, for example:
. /usr/lib/ckan/default/bin/activate
Install the ckanext-s3filestore Python package into your virtual environment:
pip install ckanext-s3filestore
- Add
s3filestore
to theckan.plugins
setting in your CKAN config file (by default the config file is located at/etc/ckan/default/production.ini
). Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:
sudo service apache2 reload
Required:
ckanext.s3filestore.aws_bucket_name = a-bucket-to-store-your-stuff
ckanext.s3filestore.region_name = region-name
ckanext.s3filestore.signature_version = signature (s3v4)
Conditional:
ckanext.s3filestore.aws_access_key_id = Your-Access-Key-ID ckanext.s3filestore.aws_secret_access_key = Your-Secret-Access-Key
Or:
ckanext.s3filestore.aws_use_ami_role = true
Optional:
# An optional path to prepend to keys
ckanext.s3filestore.aws_storage_path = my-site-name
# An optional setting to fallback to filesystem for downloads
ckanext.s3filestore.filesystem_download_fallback = true
# The ckan storage path option must also be set correctly for the fallback to work
ckan.storage_path = path/to/storage/directory
# To change the acl of the uploaded file. Default is public-read.
ckanext.s3filestore.acl = private
# To use user provided filepath and not use internal url basename on download
# Set to False to be backwards compatible to ckan file store, set to True if already using s3plugin
# If False /dataset/{id}/resource/{resource_id}/orig_download/{filename} will be available
ckanext.s3filestore.use_filename = True
# To mask the S3 endpoint with your own domain/endpoint when serving URLs to end users.
# This endpoint should be capable of serving S3 objects as if it were an actual bucket.
# The real S3 endpoint will still be used for uploading files.
ckanext.s3filestore.download_proxy = https://example.com/my-bucket/
# Cache control for signed URLs. Values are in seconds.
# 'signed_url_expiry': How long a URL is valid (default 1 hour).
# 'signed_url_cache_window': How long a URL will be reused,
# if it is not updated in the meantime (default 30 min).
# The expiry should be longer than the window (not equal);
# otherwise, a URL may expire before a new one is available.
# If either value is zero or negative, then URL caching is disabled.
ckanext.s3filestore.signed_url_expiry = 3600
ckanext.s3filestore.signed_url_cache_window = 1800
To install ckanext-s3filestore for development, activate your CKAN virtualenv and do:
git clone https://github.com/okfn/ckanext-s3filestore.git
cd ckanext-s3filestore
python setup.py develop
pip install -r dev-requirements.txt
pip install -r requirements.txt
To run the tests, do:
nosetests --ckan --nologcapture --with-pylons=test.ini
To run the tests and produce a coverage report, first make sure you have coverage installed in your virtualenv (pip install coverage
) then run:
nosetests --ckan --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.s3filestore --cover-inclusive --cover-erase --cover-tests
docker start up
docker run -it -v "pwd":/build ubuntu:bionic /bin/bash
commands before travis setup cd /build apt-get update apt-get install sudo systemd postgresql-10 git python python-pip
export PGVERSION=10 && export CKAN_BRANCH=qgov-master && export CKAN_GIT_REPO=qld-gov-au/ckan cd /build bash bin/travis-build.bash nosetests --ckan --with-pylons=subdir/test.ini --with-coverage --cover-package=ckanext.s3filestore --cover-inclusive --cover-erase --cover-tests
ckanext-s3filestore should be available on PyPI as https://pypi.python.org/pypi/ckanext-s3filestore. If that link doesn't work, then you can register the project on PyPI for the first time by following these steps:
Create a source distribution of the project:
python setup.py sdist
Register the project:
python setup.py register
Upload the source distribution to PyPI:
python setup.py sdist upload
Tag the first release of the project on GitHub with the version number from the
setup.py
file. For example if the version number insetup.py
is 0.0.1 then do:git tag 0.0.1 git push --tags
ckanext-s3filestore is available on PyPI as https://pypi.python.org/pypi/ckanext-s3filestore. To publish a new version to PyPI follow these steps:
- Update the version number in the
setup.py
file. See PEP 440 for how to choose version numbers. Create a source distribution of the new version:
python setup.py sdist
Upload the source distribution to PyPI:
python setup.py sdist upload
Tag the new release of the project on GitHub with the version number from the
setup.py
file. For example if the version number insetup.py
is 0.0.2 then do:git tag 0.0.2 git push --tags
- 0.3.0
Update from boto to boto3 Update to Ckan version 2.8+
- 0.2.0
Support for AMI Roles ACL for uploaded file can be configured don't assume that error codes are numeric fix filesystem fallback, resolves #28 set explicit ContentType on boto Put command and store the mimetype in CKAN resource table
- 0.1.1
Support for Flask-based requests
- 0.1.0
Fix downloading large files
- 0.0.9
Add populating of resources' last_modified field
- 0.0.8
Add option for fallback to local filesytem from s3
- 0.0.7
redirect always get string intead of unicode
- 0.0.6
Allow minio s3 like datastore
- 0.0.5
Add boto to install requires
- 0.0.4
Avoid exception when resources marked for clearing but not yet exist New, not yet created resources can be marked for deletion (with clear_upload) if the user cancels an upload and enters a URL instead. Check if resources have an id or if an old name is provided before trying to clear a file.
- 0.0.3
Requires CKAN 2.5+ as IUploader now in CKAN2.5
- 0.0.2
Change the resource file names to lower case
- 0.0.1
Alpha release of plugin