Skip to content

bhargavpanth/Dublin-Transportation-Consumer

Repository files navigation

Dublin Bus Data Pipeline - Consumer

What does this project do?

  • Using the modules created in here, the project establishes a pipeline to pull messages from the queue (RabbitMQ)
  • Runs classification and regression on the stream data
  • Trying to build an ensemble of multiple classification algorithms

RabbitMQ specifics

  • sudo rabbitmq-server start
  • RabbitMQ dashboard access
  • Pre-req here

Elastic Search

  • Install Elastic Search for Ubuntu - here
  • sudo systemctl enable elasticsearch.service to start Elastic Search service

Run the project

If you want to use RabbitMQ for the pipelines

  • sudo rabbitmq-server start
  • python main.py --flag=bus --host=localhost

If you want to use Kafka

  • Start the Kafka instance
  • Make the change - replace all dataframe instances to ConsumerKafka instances
  • python main.py --flag=bus --host=localhost

RabbitMQ Docker

  • Image available here

Next steps

  • Docker build this project into an image
  • Docker build container for RabbitMQ
  • Docker build containers for Elastic Search
  • Utilize GitLab CI
  • Utilize Travis for CI - Write test cases

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published