There are lots of exciting and interesting problems in analytics, such as figuring out what the right question is, selecting features, training a model, and interpreting results. But all of that presupposes a tidy data set that is suitable for analysis or training models. Industry experts all agree that data collection and preparation is roughly 3/4 of any analysis effort. Or, as Kareem Carr puts it (I'm guessing he includes data acquisition and organization in the term "cleaning"):

The title of this course is "Data Acquisition" but of course, once we get the data, we have to organize it into handy data structures and typically have to extract information from the raw data. For example, we might need to boil down a Twitter stream into a single positive or negative sentiment score for a given user. This course teaches you how to collect, organize, coalesce, and extract information from multiple sources in preparation for your analysis work. Along the way, you'll learn about networks, the internet protocols, and your own building web servers.

This course is part of the MS in Data Science program at the University of San Francisco.

INSTRUCTOR. Terence Parr. I’m a professor in the computer science and data science program departments and was founding director of the MS in Analytics program at USF (which became the MS data science program). Please call me Terence or Professor (the use of “Terry” is a capital offense).

SPATIAL COORDINATES:

• Class is held at 101 Howard in 5th floor classroom 529.
• Exams are held in the big room on the main floor. Both sections meet together.
• My office is room 607 @ 101 Howard up on mezzanine

TEMPORAL COORDINATES. Tue Aug 20 - Thu Oct 10.

• Section 01: Mon/Wed 10-11:50AM
• Section 02: Mon/Wed 1:15-3:05PM
• Exam 1: 4-5PM, Thur Sept 12 (rooms 527/529)
• Exam 2: 3:15-4:45PM, Thur, Oct 10 (last day of class, rooms 527/529)

Please note Pitch day conflicts with Thur Sept 19 so we have class Monday Sept 23 3:15 - 5:05 in room 154-156.

INSTRUCTION FORMAT. Class runs for 1:50 hours, 2 days/week. Instructor-student interaction during lecture is encouraged and we'll mix in mini-exercises / labs during class. All programming will be done in the Python 3 programming language, unless otherwise specified.

TARDINESS. Please be on time for class. It is a big distraction if you come in late.

ACADEMIC HONESTY. You must abide by the copyright laws of the United States and academic honesty policies of USF. You may not copy code from other current or previous students. All suspicious activity will be investigated and, if warranted, passed to the Dean of Sciences for action. Copying answers or code from other students or sources during a quiz, exam, or for a project is a violation of the university’s honor code and will be treated as such. Plagiarism consists of copying material from any source and passing off that material as your own original work. Plagiarism is plagiarism: it does not matter if the source being copied is on the Internet, from a book or textbook, or from quizzes or problem sets written up by other students. Giving code or showing code to another student is also considered a violation.

The golden rule: You must never represent another person’s work as your own.

If you ever have questions about what constitutes plagiarism, cheating, or academic dishonesty in my course, please feel free to ask me.

Note: Leaving your laptop unattended is a common means for another student to take your work. It is your responsibility to guard your work. Do not leave your printouts laying around or in the trash. All persons with common code are likely to be considered at fault.

USF policies and legal declarations

Students with Disabilities

If you are a student with a disability or disabling condition, or if you think you may have a disability, please contact USF Student Disability Services (SDS) for information about accommodations.

Behavioral Expectations

All students are expected to behave in accordance with the Student Conduct Code and other University policies.

Academic Integrity

USF upholds the standards of honesty and integrity from all members of the academic community. All students are expected to know and adhere to the University's Honor Code.

Counseling and Psychological Services (CAPS)

CAPS provides confidential, free counseling to student members of our community.

Confidentiality, Mandatory Reporting, and Sexual Assault

For information and resources regarding sexual misconduct or assault visit the Title IX coordinator or USFs Callisto website.

All projects will be graded with the specific input or tests given in the project description, so you understand precisely what is expected of your program. Consequently, projects will be graded in binary fashion: They either work or they do not. The only exception is when your program does not run on the grader's or my machine because of some cross-platform issue. This is typically because a student has hardcoded some file name or directory into their program. In that case, we will take off a minimum of 10% instead of giving you a 0, depending on the severity of the mistake. Please go to github and verify that the website has the proper files for your solution. That is what I will download for testing.

Each project has a hard deadline and only those projects working correctly before the deadline get credit. My grading script pulls from github at the deadline. All projects are due at the start of class on the day indicated, unless otherwise specified.

Grading standards. I consider an A grade to be above and beyond what most students have achieved. A B grade is an average grade for a student or what you could call "competence" in a business setting. A C grade means that you either did not or could not put forth the effort to achieve competence. Below C implies you did very little work or had great difficulty with the class compared to other students.

Most data you encounter will be in the form of human readable text, such as comma-separated value (CSV) files. We begin the course by studying how characters are stored in files and learning about the key data formats.

• Representing text in a computer; see also 7-bit ascii codes, unicode vs ascii in python (Day 1)
• Data pipeline project (Converting stock history from Quandl to various formats) (project) (Day 1)
  • reading delimited data; tsv, csv
  • reading/generating XML (we'll load complicated XML in TFIDF project)
  • reading/generating json
• (git intro; clone, add, commit, push, pull; see Using git revision control from Boot camp)
• Excel and CSV data (Day 2)
• PDF using pdftotext (Extracting text from Tesla's marketing brochure) (Day 2)
• HTML (Parsing Tesla's IPO prospectus) (Day 3)

There are also plenty of nontext, binary formats. You can learn more from the msds501 boot camp material for audio processing and image processing.

• Review object definition / usage (Day 3)
• Associations and dictionaries (Day 3, 4)
• Introduction to hash table construction and discussion of Search Engine Implementation (project) (Day 4)

• San Francisco police incidents word clouds (Day 4)
• (code review of sample jsontable functions, more htable discussion)
• CSV command-line kung fu (Day 5)
• Intro to information extraction from text (Day 5)
• Spacy NLP library (Day 6)
• Computing TFIDF (Day 6)
• TFIDF notebook
• TFIDF document summarization (project) (Day 6)

Now you know how to work with data files already sitting on your desk, we turn towards a study of computer networking and web infrastructure.

• Network sockets, DNS, email (Day 7)
• client/server architecture (Day 7)
• HTTP (Day 7)
• flask (Day 8)
• Launch AWS box (MSDS501 notes), AWS launch notes from Recommending Articles Project (Day 9)
• Building web servers (optional project)
• Web analytics (Day 10)
• Cookies, logging in/out (Day 11)

With an understanding of how the Internet and web works, it's time to start pulling data from various web sources. The difficulty of collecting data depends a great deal on the permissions and services available for a site or page. A good analogy is: some doors are open, some doors are closed, some doors are locked, some "doors" are not doors but reinforced steel walls.

• Pulling data from (open) REST APIs (Day 12)
  • openpayments.us (Day 12)
• Pull data from sites requiring an API key
  • Quandl stock data (Day 12)
  • OMDb movie data (Day 12)
  • Youtube (Day 13)
  • Zillow (Day 13)
• APIs requiring authentication/identification (Optional)
  • Twitter ()
  • LinkedIn ()
  • Facebook (Optional)
• Extracting data from web pages (Day 14)
  • Crawling (Day 14)
  • buzzfeed (Day 14)
  • Amazon (Day 14)
  • Scraping data from tables (Day 14)
• Selenium (Day 15)

• San Francisco police activity heat map using google map API
• Debugging with PyCharm