Python S3FileSystem.glob Examples

Example #1

def list_files(filesystem: S3FileSystem, bucket: str, s3_uri: str) -> list:
    """List files on S3 bucket on given URI."""
    # Remove leading and trailing "/"
    s3_uri = s3_uri.strip('/')
    # Find all files on s3_uri in bucket
    paths = filesystem.glob(f'{bucket}/{s3_uri}/*.parquet')

    if paths:
        return paths

    raise FileExistsError(
        f'Bucket {bucket} contains no files matching "{s3_uri}" URI')

Example #2

File: 13134_s3.py Project: tate11/intelligent-code-completion

 def glob(self, path):
     s3_path = self._trim_filename(path)
     return ['s3://%s' % s for s in S3FileSystem.glob(self, s3_path)]

Example #3

File: s3.py Project: fortizc/dask

 def glob(self, path):
     s3_path = self._trim_filename(path)
     return ['s3://%s' % s for s in S3FileSystem.glob(self, s3_path)]

Example #4

File: s3.py Project: zmyer/dask

 def glob(self, path, **kwargs):
     bucket = kwargs.pop('host', '')
     s3_path = bucket + path
     return S3FileSystem.glob(self, s3_path)