DSCI 6007: Distributed and Scalable Data Engineering

Welcome to Intro to Data Engineering!

See syllabus.md for the current syllabus.

Schedule Overview

Organized by weeks and days:
(subject to change;
see detailed schedule below)

Day	Readings	Notes	Assignment
Monday	Data Engineering Overview	1. Intro to Data Engineering 2. Intro to the Cloud	Conencting to the Cloud with Python
Tuesday	How the Internet Works	How the Web Works	Generating Reports
Thursday	Virtualization	Virtualization & Docker	Your Very Own Web Server
Friday	*NIX	Linux	Linux Intro

Day	Readings	Notes	Assignment
Monday	Introduction to Clouds	The Cloud & AWS	Move your Linux machine to the Cloud
Tuesday	Provisioning	EC2 & cron	Automate More
Thursday	I ♥ Logs	Apache Kafka	Drinking from the Firehose
Friday	Projects	Project Proposal	Proposal

Day	Readings	Notes	Assignment
Monday	Functional Programming		Fun with Toolz
Tuesday	Threading and Webscraping		Threading and Webscraping
Thursday	Intro to Multiprocessing	Multiprocessing Demonstration	Multiprocessing
Friday	Scaling Out	Distributed Computing	Embarrassingly Parallel

Day	Readings	Notes	Assignment
Monday	HDFS and MapReduce	MapReduce	Scaling Out
Tuesday	MapReduce Design Patterns	Hadoop Ecosystem	Meet MrJob
Thursday	Introduction to Spark	Apache Spark	Spark on EMR
Friday	Designing Big Data Systems	Review	Final Project Proposal

Day	Readings	Notes	Assignment
Monday	SQL Basics	Databases and SQL	Squashing Birds
Tuesday	Relational Design	Relational Database Modeling	Data Modeling Practice
Thursday	Drivers and Workers	SQL: Advanced Querying	Feeding the Elephant
Friday	Tuning SQL	Data Systems Architecture	Advanced Querying

Day	Readings	Notes	Assignment
Monday	Spark DataFrames	Spark	DataFrames
Tuesday	Programming with RDDs	Spark SQL	Spark SQL

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
0.0 - Prework		0.0 - Prework
1.1 - Course Overview		1.1 - Course Overview
1.2 - The Internet		1.2 - The Internet
1.3 - Virtualization		1.3 - Virtualization
1.4 - Linux		1.4 - Linux
2.1 - The Cloud		2.1 - The Cloud
2.2 - Deployment		2.2 - Deployment
2.3 - Big Data Architecture		2.3 - Big Data Architecture
2.4 - Projects		2.4 - Projects
3.1 - Functional Programming		3.1 - Functional Programming
3.2 - Threading		3.2 - Threading
3.3 - Multiprocessing		3.3 - Multiprocessing
3.4 - Massively Parallel Processing		3.4 - Massively Parallel Processing
4.1 - MapReduce Intro		4.1 - MapReduce Intro
4.2 - MapReduce Design Patterns		4.2 - MapReduce Design Patterns
4.3 - Spark Overview		4.3 - Spark Overview
4.4 - Proposals		4.4 - Proposals
5.1 - SQL Fundamentals		5.1 - SQL Fundamentals
5.2 - Relational Data Modeling		5.2 - Relational Data Modeling
5.3 - ETL		5.3 - ETL
5.4 - Advanced Querying		5.4 - Advanced Querying
6.1 - Spark DataFrames		6.1 - Spark DataFrames
6.2 - Spark SQL		6.2 - Spark SQL
6.3 - Review		6.3 - Review
6.4 - Spark ML Intro		6.4 - Spark ML Intro
7.1 - Spark ML and Spark Submit		7.1 - Spark ML and Spark Submit
7.2 - Spark Streaming		7.2 - Spark Streaming
8.2 - Final Project		8.2 - Final Project
Self Study		Self Study
Lab Overview.ipynb		Lab Overview.ipynb
README.md		README.md
syllabus.md		syllabus.md