ECE 720 Advanced Topics in Software Engineering and Intelligent Systems: Machine Learning System Engineering (Winter 2022)
Instructor
Lei Ma, PhD, Associate Professor, Canada CIFAR AI Chair
Email: ma.lei@ualberta.ca
Zoom (see eClass for information)
Office Hours: Wednesday 17:00-18:00
Teaching Assistant
Zhijie Wang
Email: zhijie.wang@ualberta.ca
Office: Zoom (see eClass for information)
Course Project
*Please fill out your information and project topic in this form
Course Description
*3 (fi 6) (either term, 3-0-0) Introductory and advanced topics in Machine Learning System Engineering (MLSE). Traditional software engineering, software quality assurance, and state-of-the-art software testing, analysis, etc. General-purpose machine learning and deep learning. Testing and analysis of deep learning. State-of-the-art quality assurance and engineering for machine learning-based software systems, and the future trends.
Prerequisites
This course is mostly self-contained, basic previous knowledge of traditional software engineering, testing, analysis, machine learning and deep learning are helpful, but not mandatory.
Course delivery
Lectures are delivered synchronously via zoom meeting. Lecture and hands-on materials are posted on eClass for asynchronous learning.
Course Objectives & General Content
Descriptive statement of the overall course objectives and general content to be covered in the course. This can include general information about course delivery, the importance of the course, how it fits into the overall program, etc. The data-driven AI systems(e.g., machine/deep learning) continue to achieve substantial strides in enabling cutting-edge intelligent applications, ranging from image recognition, virtual assistant (e.g., Alex, Siri), art design, autonomous vehicles, to medical diagnosis tasks. However, the development of current data-driven AI systems still lacks systematic quality assurance and engineering support in regard to the adoption of quality, security and reliability assurance standards, as well as the available mature toolchain support. The black-box nature of such AI systems poses challenges in the interpretation and understanding of the behaviors of AI systems as observed in the many recent failures of such systems. To enable safe, secure and reliable data-driven AI systems, a recent new interdisciplinary field “Machine Learning System Engineering (MLSE)” is formed and fastly growing in the past few years, which reconsiders the whole development lifecycle, providing quality assurance and engineering support for building trustworthy AI systems. The objectives of this course are to introduce the basics and current state-of-the-art techniques along this exciting direction, which includes but is not limited to:
- To introduce basic concepts of software engineering and data-driven AI (e.g., machine learning, deep learning)
- To understand the quality assurance and common techniques for “the traditional software” development process
- To learn the basics of machine learning system engineering
- To learn the basics and advanced techniques on quality assurance and common techniques of machine learning system engineering
- To understand ongoing research and industrial practices of machine learning system engineering, and the potential value and impacts.
Learning Outcomes
By the end of this course, students should be able to:
- Understand the basics of machine learning system engineering
- Know the quality assurance of traditional software development process
- Learn Automated testing of traditional software
- Understand testing and analysis of machine learning enabled software
- Get familiar with debugging and repairing of machine learning enabled software
- Know the current research trend and some industrial practices of machine learning system engineering
Course Schedule
The following table outlines the schedule for the course. We will update it as the term progresses.
| Schedule | Topics | Slides | Recordings | Recommended readings |
|---|---|---|---|---|
| Week 1 | Introduction to Machine Learning System Engineering | slides | recordings | |
| Week 2 | Introduction to Machine Learning System Engineering | slides | recordings | |
| Week 3 | Introduction to Traditional Software Engineering | slides | recordings | |
| Week 4 | Automated Software Testing I | slides | recordings | |
| Week 5 | Automated Software Testing II | slides | recordings | |
| Week 6 | MLSE Quality Assurance and ML Testing | slides | recordings | |
| Week 7 | ML Software Testing and Analysis | slides | recordings | |
| Week 8 | Reading Week (No lecture) | |||
| Week 9 | Invited Talk | |||
| Week 10 | Invited Talk | |||
| Week 11 | Invited Talk | |||
| Week 12 | Project Presentation | |||
| Week 13 | Project Presentation | |||
| Week 14 | Project Presentation |
Marking Scheme
| Activity | (A)Synchronous | Due / Scheduled | Weight |
|---|---|---|---|
| Assignment Hands-on Project 1 | Asynchronous | Posted, due on Feb 11 | 15% |
| Assignment Hands-on Project 2 | Asynchronous | See schedule in syllabus | 15% |
| Assignment Hands-on Project 3 | Asynchronous | See schedule in syllabus | 15% |
| Course Project - part 1: project proposal & literature survey | Asynchronous | See schedule in syllabus | 10% |
| Course Project - part 2: design and implementation | Asynchronous | See schedule in syllabus | 15% |
| Course Project - part 3: presentation slides | Asynchronous | See schedule in syllabus | 15% |
| Course Project - part 4: Final report | Asynchronous | See schedule in syllabus | 15% |
Term Work
All term work solutions will be posted no later than the last day of classes. All term work will be returned to students by the final day of classes, with the exception of major term work due in the last week of classes. The latter will be returned by the day of the final examination or the last day of the examination period if there is no final examination in the course as per university policy; instructors will make accommodations to return these term work. It is the responsibility of the student to pick up all their term work at the specified time and place. Any unreturned term work, shall be retained and then shredded six months after the deadline for reappraisal and grade appeals. Final examinations will be kept for one year as required by university guidelines and the Government of Alberta’s Freedom of Information and Protection of Privacy Act.
Text and References (Recommended)
- Hulten, Geoff. Building Intelligent Systems: A Guide to Machine Learning Engineering. Apress. 2018
- Molnar, Christoph. “Interpretable machine learning. A Guide for Making Black Box Models Explainable”, 2019.
- Sculley, David, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. “Hidden technical debt in machine learning systems.” In Advances in neural information processing systems, pp. 2503-2511. 2015.
- Zhang, Jie M., Mark Harman, Lei Ma, and Yang Liu. “Machine learning testing: Survey, landscapes and horizons).” IEEE Transactions on Software Engineering (2020).
- Peng, Zi, Jinqiu Yang, Tse-Hsun Chen, and Lei Ma. “A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo.” In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1240-1250. 2020.
- Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.” In Proceedings ACL, p. 4902–4912. (2020).
- Amershi, Saleema, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. “Software engineering for machine learning: A case study.” In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291-300. IEEE, 2019.
- Pei, Kexin, Yinzhi Cao, Junfeng Yang, and Suman Jana. “DeepXplore: Automated whitebox testing of deep learning systems.” In proceedings of the 26th Symposium on Operating Systems Principles, pp. 1-18. 2017.