605.256.8VL - Modern Software Concepts in Python

Module 1

Wed 05/29/24–Tue 06/04/24

Web Scraping

·       Module 1 Readings

·       Module 1 Slides Review

·       Module 1 Lecture / Readings Quiz

·       Module 1 Assignment

Module 2

Wed 06/05/24– Tue 06/11/24

Web Applications

·       Module 2 Readings

·       Module 2 Slides Review

·       Module 2 Lecture / Readings Quiz

·       Module 2 Assignment

Module 3

Wed 06/12/24– Tue 06/18/24

Databases

·       Module 3 Readings

·       Module 3 Slides Review

·       Module 3 Lecture / Readings Quiz

·       Module 3 Assignment

Module 4

Wed 06/19/24– Tue 06/25/24

Testing & Documentation

·       Module 4 Readings

·       Module 4 Slides Review

·       Module 4 Lecture / Readings Quiz

·       Module 4 Assignment

Module 5

Wed 06/26/24– Tue 07/02/24

Code Assurance & Security

·       Module 5 Readings

·       Module 5 Slides Review

·       Module 5 Lecture / Readings Quiz

·       Module 5 Assignment

Module 6

Wed 07/03/24– Tue 07/09/24

Cloud-native Development

·       Module 6 Readings

·       Module 6 Slides Review

·       Module 6 Lecture / Readings Quiz

·       Module 6 Assignment

Module 7

Wed 07/10/24– Tue 07/16/24

Cloud Computing

·       Module 7 Readings

·       Module 7 Slides Review

·       Module 7 Lecture / Readings Quiz

·       Module 7 Assignment

Module 8

Wed 07/17/24– Tue 07/23/24

Data Preparation & Statistics

·       Module 8 Readings

·       Module 8 Slides Review

·       Module 8 Lecture / Readings Quiz

·       Module 8 Assignment

Module 9

Wed 07/24/24– Tue 07/30/24

Data Preparation & Models

·       Module 9 Readings

·       Module 9 Slides Review

·       Module 9 Lecture / Readings Quiz

·       Module 9 Assignment

Module 10

Wed 07/31/24– Tue 08/06/24

Data Visualization

·       Module 10 Readings

·       Module 10 Slides Review

·       Module 10 Lecture / Readings Quiz

·       Module 10 Assignment

Module 11

Wed 08/07/24– Tue 08/13/24

Machine Learning Operations (MLOps)

·       Module 11 Readings

·       Module 11 Slides Review

·       Module 11 Lecture / Readings Quiz

·       Module 11 Assignment

Module 12

Wed 08/14/24– Tue 08/20/24

Concurrency & Scale

·       Module 12 Readings

·       Module 12 Slides Review

·       Module 12 Lecture / Readings Quiz

·       Module 12 Assignment

1

Web Scraping[GR1] [DLG[U2] 

We begin by welcome to the course and introducing the lecturers.

We then establish course expectations and move through the syllabus, course outline, office hours, grading policy, and course delivery tools.

Next, we discuss web scraping at a high level and teach handling URLs in Python with urllib3.

Afterwards, we begin a discussion of reading data scraped from the web with built-in string methods and regex string methods.

We then conclude with a discussion of BeautifulSoup to interact/parse HTML.

An extension (not tested) into mechanicalSoup to automate work carried out using BeautifulSoup is offered.

By the end of this module, you will be able to:

1.1 – Course Understanding: Maintain a clear understanding of our course structure and key documentation.

1.2 –  URL Handling: manage http request-responses in python with urllib3

1.3 – Data Extraction: Extract data from web pages and utilize regular expressions

1.4 – HTML Object Parsing: Apply BeautifulSoup for effective data scraping

1.5  - Github Version Control: Store and version control assignment code.

1A: Introduction to Web Scraping (1.1-1.4) – a voiceover PowerPoint presentation.

1B: Real Python (1.2-1.4) – Select readings

1C: Github Version Control (1.5) – Instruction setup guide.

Assessment

·      Students complete a web scraping assignment that requires them to pull data from grad cafe, search for necessary variables, and store data as JSON files, and then upload to Github. (1.1-1.5)

·      Students complete a short quiz covering their understanding of course policy. (1.1)

·      Students complete a short quiz covering their understanding of web scraping in python. (1.2-1.4)

https://realpython.com/python-web-scraping-practical-introduction/

https://realpython.com/beautiful-soup-web-scraper-python/

Github Version Control Guide

2

Web Applications[GR3] [DLG[U4] 

We begin by discussing web applications, server-client architectures, HTTP request-response, and static vs dynamic applications.

Next, we discuss building web applications with Flask.

Afterwards, we provide an example of deploying web applications on Amazon Web Services.

An extension (not tested) into Flask architecture tool, Blueprint is offered in closing.

2.1 – Web Application Architecture: Gain a clear understanding of the language-agnostic underlying requirements of web application systems.

2.2 – Web Application Construction: Build web applications in python using flask, add HTML templates, JavaScript, and CSS files.

2.3  – Local Web Application Testing: Test / stage applications locally, build multiple page websites.

2.4  – Cloud Web Application Deployment: Learn to deploy web applications using cloud services.

2A: Introduction to Web Applications (2.1-2.4) – a voiceover PowerPoint presentation.

2B: Real Python (2.1-2.3) – Select Readings

2C: Student AWS account (2.4) – Instruction setup guide.

Assessment

·      Students complete a web application development assignment that requires them build a personal website using flask, and then deploy the application to a Student AWS account.  (2.1-2.4)

·      Students complete a short quiz covering their understanding of web application development and deployment. (2.1-2.4)

https://realpython.com/flask-project/

https://realpython.com/flask-javascript-frontend-for-rest-api/

AWS Account Instructions

3

Databases[GR5] [DLG[U6] 

We begin by reflecting upon the application data layer – looking at the simplified layers used in M1 and M2.

We then cover different application tiers: presentation, application, and data. We discuss the data layer and cover the need for databases.

Next, we cover relational databases, structuring data tables, setting up a local PostgreSQL database, and loading JSON files into a relational database.

Afterwards, we use Psycopg3 to access relational databases and carry out SQL commands to search the database.

An extension (not tested) into SQLAlchemy2 for pure SQL development.

3.1 – Application Tiers: Gain a clear understanding of the typical tiers that make up the architecture of applications.

3.2 – Relational Database Setup: learn to set up, load data, and access a local relational database.

3.3 – Search Databases for Relevant Data: Use Psycopg3 to search and manipulate relational databases in Python.

3.4 – Git Branching: Learn to develop in partnership through branching, merging, and commit history.

3A: Introduction to Databases (3.1-3.3) – a voiceover PowerPoint presentation.

3B: Real Python (3.1-3.3)  

3C: Git branching, merging, and commit demo.

Assessment

·      Students complete a database assignment that requires them to take that json data developed in week 1 and load it into a PostgreSQL database. The connect to it using Psycopg3 to answer analysis questions. Students must develop code using branches and show commit history. (3.1-3.4)

·      Students complete a short quiz covering their understanding of database development and deployment. (3.1-3.3)

https://realpython.com/prevent-python-sql-injection/

Git branching, merging, and commit demo.

4

Testing & Documentation

We begin by recognizing that to deploy reliable code that behaves in expected ways, we will need to perform testing.

We then cover unit tests and efficient unit test development with pytest.

Next, we cover how to parameterize unit tests, manage fixtures, and structure dependencies.

Afterwards, we cover categorizing tests and combining tests.

We move to generating code documentation with sphinx.

An extension (not tested) into hosting documentation is offered in closing.

4.1 – Handling and Testing Unexpected Inputs: Learn when an interaction could break code / cause unexpected behavior.

4.2 – Testing Code: Learn to address potential code issues by writing scalable, parameterized, unit tests in python.

4.3 – Documenting Code: Learn to create and release standardized documentation about your codebase.

4A: Introduction to Testing & Documentation (4.1-4.3) – a voiceover PowerPoint Presentation.

4B: Real Python (4.1-4.3)

4C: Sphinx exemplar walkthrough (4.3)

Assessment

·      Students develop a pygame simulation and character development, then they test their game behavior using pytest. They then provide sphinx documentation for their application. (4.1-4.3)

·      Students complete a short quiz covering their understanding of test development and documentation. (4.1-4.3)

https://realpython.com/courses/python-sphinx/

https://realpython.com/pytest-python-testing/

Sphinx demo walkthrough / practice creation.

5

Code Assurance & Security

We begin by recognizing that code is written once and read many times. Therefore, having standardized, well-written code is important.

We then introduce linters to help automate the process of ensuring code is standardized. We teach pylint to match google code standard.

Next, we cover the need to consider code security performance. We discuss ways code in deployment could be manipulated to create unexpected behaviors.

We teach snyk as an emerging tool to help automate the process of considering code vulnerabilities.

5.1 – Importance of Standardized, Readable Code: Learn how and why writing code is a standardized way is important – consider several code standards.

5.2 – Automate Code Standards: Using linters to help test how closely your code matches appropriate standards.

5.3 – Code Security Vulnerabilities: Consider potential ways an adversary can manipulate your application’s source code.

5.4 – Security Evaluation Tooling: Learn how to use emerging tool, snyk to help automate code vulnerability testing.

5A: Introduction to Code Assurance & Security (5.1-5.4) – a voiceover PowerPoint Presentation.

5B: Real Python (5.1-5.2)

5C: Snyk Blog (5.3-5.4)

Assessment

·      Students extend upon their pygame simulation and character development, they must now also lint their code to 10/10 standards and test for code vulnerabilities with snyk. (5.1-5.4)

·      Students complete a short quiz covering their understanding of code assurance and security. (5.1-5.4)

https://realpython.com/courses/writing-cleaner-python-code-pylint/

https://snyk.io/blog/getting-started-snyk-for-secure-python-development/

6

Cloud Native Development

We begin by discussing the problems created when trying to deploy the same application into different environments.

We introduce docker as a way to seamlessly deploy applications anywhere through containerization.

We then discuss the structure of dockers key components: dockerfiles, images, containers, and dockerhub.

As an extension (not tested) into docker compose for multi-container applications is provided.

6.1 – Deploy Anywhere Code: How to separate your code from underlying system through virtualization.

6.2 – Containerization: Use docker to containerize applications, make creation repeatable through dockerfiles, and publish environment images to dockerhub.

6.3 – Standard Virtual Environments: Take advantage of pre-built base images for applications.

6A: Introduction to Cloud Native Development (6.1-6.3) – a voiceover PowerPoint Presentation.

6B: Git Connected Reading (6.1-6.2)

6C: Docker Reading (6.3)

Assessment

·       Students complete an assignment dockerizing their pygame simulation and character development application. They then deploy to cloud and make accessible from their website. (6.1-6.3)

·       Students complete a short quiz covering their understanding of cloud native development. (6.1-6.3)

https://levelup.gitconnected.com/dockerizing-a-flask-application-with-a-postgres-database-b5e5bfc24848

https://docs.docker.com/docker-hub/

7

Cloud Computing[GR7] [DLG[U8] 

We begin with a discussion of on-premises vs cloud / hybrid workflows and why cloud has become a powerful emerging construct for computing resource management.

We then introduce several cloud providers, and indicate that many of the fundamental capabilities are comparable.

Next, we focus in on application construction using AWS – focusing on interaction using Python’s boto3 library.

We end with a discussion of sagemaker, a pre-set Data Science environment offered by AWS.

An extension (not tested) into cloud resource/cloud spend management is provided.

7.1 – Deployment Decision Tradeoff: how to decide whether to deploy your application on-prem or in the cloud.

7.2 – Cloud Providers and Services: overview cloud providers and similar core offerings.

7.3 – Use Python to interact with cloud services: dive into boto3 for AWS interaction.

7.4 – Data Science in Cloud: introduction into SageMaker for Data Science development.

7A: Introduction to Cloud Computing (7.1-7.4) – a voiceover PowerPoint Presentation.

7B: AWS Reading (7.1-7.2)

7C: Real Python Reading (7.3-7.4)

Assessment

·       Students complete an assignment focused on cloud service interaction using boto3. (7.1-7.3)

·       Students complete a short quiz covering their understanding of Cloud Computing. (7.1-7.4)

https://aws.amazon.com/education/awseducate/

https://realpython.com/python-boto3-aws-s3/

8

Data Preparation & Statistics[GR9] [DLG[U10] 

We first discuss the need to structure aggregated data and prepare for analysis.

We then discuss data preparation for common use cases: database storage, model development, downstream applications.

We practice creating numPy arrays, and shaping, filtering, ordering, and aggregating using numPy.

We then move to using Pandas for tabular dataFrame data displays /organization of data.

We demonstrate how to retrieve, access, sort, and modify pandas object.

We end by discussing advanced dataframe/numpy array manipulation using pre-build methods from SciPy.

8.1 – Data Preparation: Using numPy to interact with with n-dimensional data.

8.2 – Data Manipulation: Using numPy for array shaping, filtering, ordering, and aggregating.

8.3 – Tabular Data Manipulation: create, retrieve, access, sort, and modify data using Panda DataFrames.

8.4 – Advanced Manipulation: Applying SciPy and NumPy functions to Panda DataFrames.

8A: Introduction to Data Preparation & Statistics (8.1-8.3) – a voiceover PowerPoint Presentation.

8B: Real Python Reading (8.1-8.3)

Assessment

·       Students complete an image-based data preparation and statistics assignment focused on developing deep understanding and familiarity with numPy for statistics. (8.1-8.3)

·       Students complete a short quiz covering their understanding of Data Preparation & Statistics. (8.1-8.3)

https://realpython.com/numpy-tutorial/

https://realpython.com/pandas-dataframe/

9

Data Preparation & Models[GR11] [DLG[U12] [DLG[U13] 

We extend upon our data preparation covered for statistics and data cleaning in M8, with a focus on model management.

We continue our discussion of advanced numPy, particularly focusing on indexing and slicing.

We then use scikit-learn to develop training / test sets as a needed pre-processing step.

Afterward, we extend our data preparation work to focus on text data / vectorization.

We then look at the complete process of model training pipeline development, with a systems design perspective.

We end with a discussion of model/data-enabled software development techniques. 

9.1 – Advanced Manipulation: Applying Numpy and scikit-learn for data preparation / model training setup.

9.2 – Text Data Preparation: Applying data preparation methods to text data.

9.3 – Model Training Pipelines: key steps in model training pipelines and how to organize / test model training code.

9.4 – Applications with Statistical Models: Testing software applications that include statistical models.  

9A: Introduction to Data Preparation & Models (9.1-9.4) – a voiceover PowerPoint Presentation.

9B: Real Python Reading (9.1-9.4)

Assessment

·       Students complete a text-based data preparation and models assignment focused on developing deep understanding and familiarity with numPy and tensors for model development. (9.1-9.4)

·       Students complete a short quiz covering their understanding of Data Preparation & Models (9.1-9.4)

https://realpython.com/lessons/indexing-and-slicing/

https://realpython.com/courses/splitting-datasets-scikit-learn-train-test-split/

10

Data Visualization

We start by discussing the need to deploy data visualizations to help with analysis and understanding.

We then look at common visualization library, MatplotLib.

We discuss MatplotLib object hierarchy, plots, subplots, figures, and then turn to services like chatGPT to automat MatplotLib code development.

Afterwards, we look at making plots interactive with plotly.

We end with an example of building data visualization applications in python with streamlit.

10.1 – Data Visualization Tools: Learn to build data visualizations with MatplotLib.

10.2 – Animated Data Visualizations: Use Plotly to build interactive data visualizations.

10.3 –Data Visualization Applications: Learn to build data visualization applications with streamlit.

10A: Introduction to Data Visualization (10.1-10.4) – a voiceover PowerPoint Presentation.

10B: Real Python Reading (10.1-10.4)

Assessment

·       Students complete a data visualization project that highlights how to visualize outputs from models using matplotlib, streamlit, and plotly. (10.1-10.4)

·       Students complete a short quiz covering their understanding of Data Preparation & Statistics (10.1-10.4)

https://realpython.com/python-matplotlib-guide/

https://realpython.com/python-dash/

11

Machine Learning Operations (MLOps)

We start by discussing the need to update, adapt, test, and redeploy model-based applications. We introduce the term MLOps to describe this process.

To fulfill this model deployment architecture, we introduce MLFlow, a popular MLOps tool.

We then introduce MLFlow Server, and cover model setup, data setup, model evaluation, logging, model registries, and model deployment.

We end with a conversation on data persistence – managing data storage, versioning, and determining when / what data to delete.

11.1 – MLOps Concepts: Learn to automate and simplify ML workflows and deployments.

11.2 MLOps Implementation: Learn MLFlow as a tool to carry out MLOps processes.

11.3 Data Persistence: Learn to manage data storage, versioning, and management.

11A: Introduction to Machine Learning Operations (11.1-11.3) – a voiceover PowerPoint Presentation.

11B: Medium Reading (11.1-11.2) – access copied /downloaded.

11C: AWS Data Persistence Reading (11.3)

Assessment

·       Students use their text model from week 10 and look at deploying it over MLflow to complete this assignment. (11.1-11.3)

·       Students complete a short quiz covering their understanding of Data Preparation & Statistics (11.1-11.3)

https://medium.com/analytics-vidhya/manage-your-machine-learning-lifecycle-with-mlflow-in-python-d678d5f3c682

https://docs.aws.amazon.com/prescriptive-guidance/latest/modernization-data-persistence/enabling-patterns.html

12

Concurrency & Scale

We start with a discussion of concurrency, parallelization, and threading.

We then put together an example of multi-threading, managing threads in larger application.

We next look at applications of concurrency to big data problems.

We introduce dask as a python tool to handle concurrency for big data.

We end with a discussion of thread safety.

12.1 Concurrency & Parallelization: Understanding of key concepts.

12.2 Concurrency and Big Data: Use Dask to manage concurrency in Python for big data problems.

12.3:  Thread Safety: Understand and mitigate race conditions in multi-threaded code.

12A: Introduction to Concurrency & Scale (12.1-12.4) – a voiceover PowerPoint Presentation.

12B: Real Python Reading (12.1-12.2)

12C: DataCamp Reading: (12.2-12.3)

Assessment

·       Students complete a multi-threading homework assignment over a large data statistic problem and then use dask to parallelize. (12.1-12.3)

·       Students complete a short quiz covering their understanding of Data Preparation & Statistics (12.1-12.3)

https://realpython.com/python-concurrency/

https://www.datacamp.com/courses/parallel-programming-with-dask-in-python

Excellent

(90-100)

Good

(80-89)

Satisfactory

(70-79)

Unsatisfactory

(70>)

Program Correctness

60%

Code compiles successfully and produces correct results for all inputs. The program does not produce any runtime issues and makes use of relevant concepts from the module

Code compiles successfully and produces mostly correct results for all inputs. The program does not produce any runtime issues. Borrows from concepts beyond the curriculum.

Code compiles successfully and produces some correct results for all inputs. Program may encounter a runtime error.

Code does not compile successfully, produces multiple incorrect results, and/or suffers from multiple runtime errors.

Code Quality/Style

15%

Cleanly written and follows Google Python Style Guidelines. Nicely organized, makes good use of whitespace, and appropriately named variables, methods, and classes. Pytest 10/10 compliant (after taught).

Generally, follows Google Python Style Guidelines. Mostly well-organized, slightly over- or under- used whitespace. Not pytest 10/10 compliant (after taught).

Misses most Google Python Style Guidelines. Excessive use of whitespace. Poorly chosen names for variables, methods, and classes.

Omits Google Python Style Guidelines. Significant misuse (or omission) of whitespace or comments. Arbitrarily named variables. Code is not easy to read/follow.

Documentation

15%

All relevant code is commented in appropriate detail. Different types of comments are used appropriately. readme.txt is provided and gives a clear and concise description of the approach taken. Sphinx documentation html available and error free (after taught).

Most relevant code is commented in appropriate detail. Too many/too few comments provided. readme.txt is mostly clear and the approach taken is mostly correct. Sphinx documentation is available with minor errors (after taught).

Multiple necessary comments are missing. Comments are routinely too long/too short. readme.txt is somewhat clear or the approach taken is not correct. Minimal Sphinx documentation is available or is with major errors (after taught).

Few to no comments are provided. Comments contain little to not useful information. readme.txt is missing or the approach taken is incorrect for the problem to be solved. No sphinx documentation or non-compliable sphinx documentation (after taught).

Assignment Specifications

10%

All required files are submitted in the required format, named correctly, and program outputs are in the proper format. All Assignment instructions have been followed.

Minor errors in requirement files or files submitted in an incorrect format, named incorrectly, or small errors in the program outputs.

Major errors in required files or files submitted in an incorrect format, named incorrectly, or major errors in the program outputs.

Submission does not meet most of the submission requirements or formats. Outputs are unintelligible.