ECE 720 Advanced Topics in Software Engineering and Intelligent Systems: Machine Learning System Engineering (Winter 2022)

Instructor


Lei Ma, PhD, Associate Professor, Canada CIFAR AI Chair

Email: ma.lei@ualberta.ca

Zoom (see eClass for information)

Office Hours: Wednesday 17:00-18:00

Teaching Assistant


Zhijie Wang

Email: zhijie.wang@ualberta.ca

Office: Zoom (see eClass for information)

Course Project


*Please fill out your information and project topic in this form

Course Description


*3 (fi 6) (either term, 3-0-0) Introductory and advanced topics in Machine Learning System Engineering (MLSE). Traditional software engineering, software quality assurance, and state-of-the-art software testing, analysis, etc. General-purpose machine learning and deep learning. Testing and analysis of deep learning. State-of-the-art quality assurance and engineering for machine learning-based software systems, and the future trends.

Prerequisites


This course is mostly self-contained, basic previous knowledge of traditional software engineering, testing, analysis, machine learning and deep learning are helpful, but not mandatory.

Course delivery


Lectures are delivered synchronously via zoom meeting. Lecture and hands-on materials are posted on eClass for asynchronous learning.

Course Objectives & General Content


Descriptive statement of the overall course objectives and general content to be covered in the course. This can include general information about course delivery, the importance of the course, how it fits into the overall program, etc. The data-driven AI systems(e.g., machine/deep learning) continue to achieve substantial strides in enabling cutting-edge intelligent applications, ranging from image recognition, virtual assistant (e.g., Alex, Siri), art design, autonomous vehicles, to medical diagnosis tasks. However, the development of current data-driven AI systems still lacks systematic quality assurance and engineering support in regard to the adoption of quality, security and reliability assurance standards, as well as the available mature toolchain support. The black-box nature of such AI systems poses challenges in the interpretation and understanding of the behaviors of AI systems as observed in the many recent failures of such systems. To enable safe, secure and reliable data-driven AI systems, a recent new interdisciplinary field “Machine Learning System Engineering (MLSE)” is formed and fastly growing in the past few years, which reconsiders the whole development lifecycle, providing quality assurance and engineering support for building trustworthy AI systems. The objectives of this course are to introduce the basics and current state-of-the-art techniques along this exciting direction, which includes but is not limited to:

  • To introduce basic concepts of software engineering and data-driven AI (e.g., machine learning, deep learning)
  • To understand the quality assurance and common techniques for “the traditional software” development process
  • To learn the basics of machine learning system engineering
  • To learn the basics and advanced techniques on quality assurance and common techniques of machine learning system engineering
  • To understand ongoing research and industrial practices of machine learning system engineering, and the potential value and impacts.

Learning Outcomes


By the end of this course, students should be able to:

  1. Understand the basics of machine learning system engineering
  2. Know the quality assurance of traditional software development process
  3. Learn Automated testing of traditional software
  4. Understand testing and analysis of machine learning enabled software
  5. Get familiar with debugging and repairing of machine learning enabled software
  6. Know the current research trend and some industrial practices of machine learning system engineering

Course Schedule

The following table outlines the schedule for the course. We will update it as the term progresses.


Schedule Topics Slides Recordings Recommended readings
Week 1 Introduction to Machine Learning System Engineering slides recordings  
Week 2 Introduction to Machine Learning System Engineering slides recordings  
Week 3 Introduction to Traditional Software Engineering slides recordings  
Week 4 Automated Software Testing I slides recordings  
Week 5 Automated Software Testing II slides recordings  
Week 6 MLSE Quality Assurance and ML Testing recordings slides  
Week 7 ML Software Testing and Analysis (Continued) slides recordings  
Week 8 Reading Week (No lecture)      
Week 9 ML Software Debugging, Interpretation, Repairing      
Week 10 Invited Talk      
Week 11 Invited Talk      
Week 12 Project Presentation      
Week 13 Project Presentation      
Week 14 Project Presentation      

Marking Scheme


Activity (A)Synchronous Due / Scheduled Weight
Assignment Hands-on Project 1 Asynchronous Posted, due on Feb 11 15%
Assignment Hands-on Project 2 Asynchronous See schedule in syllabus 15%
Assignment Hands-on Project 3 Asynchronous See schedule in syllabus 15%
Course Project - part 1: project proposal & literature survey Asynchronous See schedule in syllabus 10%
Course Project - part 2: design and implementation Asynchronous See schedule in syllabus 15%
Course Project - part 3: presentation slides Asynchronous See schedule in syllabus 15%
Course Project - part 4: Final report Asynchronous See schedule in syllabus 15%

Term Work

All term work solutions will be posted no later than the last day of classes. All term work will be returned to students by the final day of classes, with the exception of major term work due in the last week of classes. The latter will be returned by the day of the final examination or the last day of the examination period if there is no final examination in the course as per university policy; instructors will make accommodations to return these term work. It is the responsibility of the student to pick up all their term work at the specified time and place. Any unreturned term work, shall be retained and then shredded six months after the deadline for reappraisal and grade appeals. Final examinations will be kept for one year as required by university guidelines and the Government of Alberta’s Freedom of Information and Protection of Privacy Act.


  • Hulten, Geoff. Building Intelligent Systems: A Guide to Machine Learning Engineering. Apress. 2018
  • Molnar, Christoph. “Interpretable machine learning. A Guide for Making Black Box Models Explainable”, 2019.
  • Sculley, David, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. “Hidden technical debt in machine learning systems.” In Advances in neural information processing systems, pp. 2503-2511. 2015.
  • Zhang, Jie M., Mark Harman, Lei Ma, and Yang Liu. “Machine learning testing: Survey, landscapes and horizons).” IEEE Transactions on Software Engineering (2020).
  • Peng, Zi, Jinqiu Yang, Tse-Hsun Chen, and Lei Ma. “A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo.” In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1240-1250. 2020.
  • Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.” In Proceedings ACL, p. 4902–4912. (2020).
  • Amershi, Saleema, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. “Software engineering for machine learning: A case study.” In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291-300. IEEE, 2019.
  • Pei, Kexin, Yinzhi Cao, Junfeng Yang, and Suman Jana. “DeepXplore: Automated whitebox testing of deep learning systems.” In proceedings of the 26th Symposium on Operating Systems Principles, pp. 1-18. 2017.