SwiftParallelizer

A way to upload large objects in parallel by splitting into small chunks to Openstack swift using Openstack Marconi as the queue transport service.

Usage

Install Openstack Swift SAIO

Install Openstack Marconi with MongoDB controller (In-memory). This can be done by modifying dbpath to a tmpfs file system in linux

Put files inside the objects directory

$ ./parallelize.sh "filename" "split size" "threadcount"

$ ./parallelize.sh rollercoaster.jpg 100k 10

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
Objects		Objects
filesplit		filesplit
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
__init__.py		__init__.py
connection.py		connection.py
create_container.py		create_container.py
deleteobject.py		deleteobject.py
files_into_queue.py		files_into_queue.py
get_connection.py		get_connection.py
getobject.py		getobject.py
listobjects.py		listobjects.py
marconi_queuing.py		marconi_queuing.py
multiprocessing_import_worker_queue_in.py		multiprocessing_import_worker_queue_in.py
multiprocessing_import_worker_queue_out.py		multiprocessing_import_worker_queue_out.py
multiprocessing_queue_in.py		multiprocessing_queue_in.py
multiprocessing_queue_out.py		multiprocessing_queue_out.py
multiprocessing_test.py		multiprocessing_test.py
objectstore_into_queue.py		objectstore_into_queue.py
parallelize.sh		parallelize.sh
putobject.py		putobject.py
read_xauth.py		read_xauth.py
rollercoaster.jpg		rollercoaster.jpg