Repository contains labs and Final Project
- Accessing cloud resources
- Ability to copy files to HDFS
- Ability to run an existing program in Hadoop using Python
- Structure of Hadoop programs
- Wow Hadoop scheduling interacts with HDFS
Construct URL-Lister hadoop program
- Select proper method for join
- Constructing more complex programs using ChainMapper
- Extending Hadoop: Pig, Hive
Construct Patent application
- The assumptions & problems with Hadoop
- In-memory computing: Spark & RDD
Construct patent application using PySpark
- Computing Abstractions and realization in DC
- CPU, Memory, Storage, Network, Processes
- Benefits & Challenges of Virtualization
- Managability, programability, utility modle
- Realization of computing abstractions in DC
- VM, VPC / Network, Block store, object store, container
Use libcloud to create VM with specific software, SSH keys, etc
- Common components of many data center scale applications
- Message marshalling
- RPC / API interfaces
- Message queues
- Databases
- Key-Value Stores
- Distributed Lock Manager
- Virtual Machines
- Software Defined Networks
- Service oriented architecture In Depth:
- Message Marshalling
- REST API interfaces
Existing current libcloud VM to include REST API
- Message Queues
- RabbitMQ
- ZeroMQ
- Distributed Lock Managers
- Zookeeper
- Etcd
- Key-Value stores - Redis, others
- Scalable Databases
- Challenges in scaling
License plate service with discovery, REST API, database
- Software virtualization
- Hardware virtualization
- Paravirtualization
- SDN
- Process based virtualization - Docker containers
- cgroups / namespaces
- Security model, benefits, isolation
- Kubernetes
- Software structure
- Using kubernetes
- Assignment:
- Converting ALPR to Kubernetes
- Advanced Kubernetes
- Function as service (lambda, etc)
- Warehouse scale computing design
- Power efficiency of warehouse scale computing
- Projects drafts submitts