多模態影像資料處理
Multi-Modality Image Processing, MMIP
| 節 | 週三 |
|---|---|
5 13:20–14:10 | 多模態影像資料處理 CM217（歸仁） 3 節連堂 |
6 14:20–15:10 | |
7 15:30–16:20 |
* 根據陽明交大上課時間表所列
This course is co-designed by Prof. Chih-Chung Hsu and the industry team Kanai Intelligence (Eric Cheng) as an industry-driven, hands-on multimodal AI course. Unlike traditional courses that stop at theory or stay within a single model, it takes students from the fundamentals of computer vision and deep learning all the way to Multimodal Large Language Models (Multimodal LLMs) that understand images, text, and other signals at once, culminating in building a deployable, monitorable, and safeguarded Multi-Modal Agentic System. The content maps end-to-end onto the skills the industry actually needs to build real multimodal AI products. The course emphasizes learning by doing: every week is paired with hands-on Google Colab practice, and students are encouraged throughout to co-develop code with generative AI tools (Codex, Claude Code, ChatGPT, Gemini, etc.), cultivating real-world AI-assisted engineering skills. Topics span image analysis, deep learning and CNNs, NLP and Transformers, multi-head and multimodal models, Flask front-/back-end web development, Retrieval-Augmented Generation (RAG), Function Calling and Agent Skills, dynamic code generation, system attack and defense, and cloud deployment with usage/cost monitoring. Learning objectives: 1. Understand the core principles and technical evolution from computer vision and deep learning to multimodal LLMs. 2. Train, integrate, and evaluate multimodal deep learning models that process images and text simultaneously. 3. Integrate AI models into front-/back-end websites to build interactive multimodal application services. 4. Design multimodal agentic systems with RAG, Function Calling, and Agent Skills. 5. Assess system security, design Moderation and Guardrail mechanisms, and complete cloud deployment with cost/performance monitoring. 6. Become proficient at co-developing with generative AI tools, building engineering skills aligned with the pace of industry.
Basic Python programming is sufficient. Having taken (or concurrently taking) introductory linear algebra, probability, and statistics is a plus. No prior deep learning background is required—the course builds up from computer vision and deep learning fundamentals. Students must bring their own laptop and be able to use Google Colab.
無備註
1. Please always bring a laptop; the course is paired with Google Colab throughout. 2. Co-developing code with generative AI tools is encouraged: Codex, Claude Code, ChatGPT, Gemini, Antigravity, etc. 3. Using GitHub is strongly recommended—push your weekly work to a personal repository as the basis for assignment submission and the final showcase. 4. A blended format of theory + hands-on practice + projects, with midterm and final project presentations. 5. Industry co-teaching: co-taught by Eric Cheng of Kanai Intelligence, sharing real-world industry experience. Contact: Eric Cheng | Email: ccc@erixnet.com | Website: erixnet.com
Hands-on assignments: 14 total, 5% each = 70% (you are strongly encouraged to push your weekly work to a personal GitHub repository as the basis for submission). Midterm report (Multi-Modal Discriminative System): 10%. Final report (Multi-Modal Agentic System): 20%. Total: 100%.
教師未提供此項資料
| 週次 | 主題 |
|---|---|
| 第 1 週 | Topic: Course introduction + Introduction to Computer Vision Assignment 01: Use OpenCV to perform computer vision analysis on images Supplement: Sharing experiences of co-developing code with generative AI tools |
| 第 2 週 | Topic: Deep Learning Assignment 02: Train and analyze an MLP for image classification |
| 第 3 週 | Topic: Convolutional Neural Networks (CNN) Assignment 03: Train and analyze a CNN for image classification |
| 第 4 週 | Topic: Natural Language Processing (NLP) and Transformers Assignment 04: Train and analyze an NLP model Supplement: Vision Transformer |
| 第 5 週 | Topic: Multi-Head deep learning models Assignment 05: Train a multi-head DL model that processes images and text simultaneously |
| 第 6 週 | Topic: Flask web development + Midterm project milestone Assignment 06: Build a website integrating a front-end UI with a back-end AI model Supplement: Using Antigravity, Codex, Claude Code to assist development |
| 第 7 週 | Topic: How multimodal LLMs work, using Google Gemini as an example Assignment 07: Add a multimodal LLM to the website |
| 第 8 週 | Midterm project presentation (Multi-Modal Discriminative System) |
| 第 9 週 | Topic: Structured Output LLMs Assignment 08: Parse input information via a multimodal LLM |
| 第 10 週 | Topic: Retrieval-Augmented Generation (RAG) Assignment 09: Build an FAQ customer-service AI chatbot Supplement: Using Antigravity, Codex, Claude Code to assist development |
| 第 11 週 | Topic: Designing Function Calling and Agent Skills Assignment 10: Build a multimodal interactive system with LangGraph Supplement: Using Antigravity, Codex, Claude Code to assist development |
| 第 12 週 | Topic: Dynamic Code Generation and Agent Skills Assignment 11: Implement an AI agent that generates its own code |
| 第 13 週 | Topic: Flask web development + Final project milestone Assignment 12: Submit a plan for integrating an Agent into the midterm project |
| 第 14 週 | Topic: Attacks on and defense of LLMs and agentic systems Assignment 13: Design Moderation and Guardrail mechanisms |
| 第 15 週 | Topic: Monitoring and deployment of multimodal agentic systems Assignment 14: Compute token usage and the time spent on each API call; log every use and store it in a database Supplement: Deploy the system to Google Cloud Platform |
| 第 16 週 | Final project presentation (Multi-Modal Agentic System) |
No single textbook is required. The course relies on lecture notes, Google Colab example code, and online documentation. Reference resources include the official docs for OpenCV, PyTorch, Hugging Face Transformers, LangGraph, and the Google Gemini API, as well as the course GitHub repository.
- 地點
- By arrangement (online via Google Meet available).
- 時間
- After class, or by appointment via email.
- 聯絡方式
- Eric Cheng | Email: ccc@erixnet.com | Website: erixnet.com