|
|
|
|
|
The concept of Image-Based Rendering (IBR) was introduced in the early 1990s to refer to a new paradigm in computer graphics image synthesis. Instead of using traditional geometric models and material properties fed in as inputs to a rendering system (much like what you would do in CSE 167 or 168), one goes directly from input images (photorealistic by definition if taken from real photographs) to new images. The original problem is often called view synthesis today, where we take a number of input images and enable synthesis of new views from different directions. There are numerous applications in e-commerce, scene acquisition and virtual reality. The original image-based rendering papers kicked off a great deal of interest in computer vision and graphics, providing many new directions for the field.
Image-based rendering has remained an active area for the 30+ years since. Along the way, new methods for large-scale geometric reconstruction have been developed, ultimately leading to photorealistic renderings of the full earth such as in Streetview or Google Earth. In the past five years, there has been a greatly renewed interest in view synthesis, with tens of thousands of papers a year, initiated by work on volumetric scene representations and neural radiance fields from the instructor, and newer developments like Gaussian splatting. These new developments are also critical to generative AI, and can leverage those models, as they often provide a bridge from 2D GenAI training to 3D and video synthesis.
This course seeks to provide a comprehensive overview of image-based rendering from relevant background to the early seminal papers, to the exciting new developments currently ongoing, and bring students up to the frontiers of the field. We expect there to be something for everyone in this course. Even if you are currently actively researching modern view synthesis, the course should cover important historical background and provide a broader perspective of the area. For those new to the field, it provides an overview of a very relevant topic. Please note that we assume a background in basic computer graphics at the level of CSE 167 (you are encouraged to take my online course on edx and receive clearance if you have not already done so). Similarly, while not required, we encourage you to take my online CSE 168 course on edx if you have not already done so (please let me know if you have any difficulty finding these courses).
While knowledge of computer vision or basic deep learning is not formally required, it certainly doesn't hurt, especially if pursuing a topic on the later parts of the course. The course will provide a brief high-level introduction, but we don't have time to get into all of the details. Note that almost all papers within the last 5 years on image-based rendering have involved some form of learning or at least optimization, and we encourage taking other learning/optimization courses in the department. We are hoping for this course to have access to the instructional machine learning cluster, and we may have some older GPUs available, please ask. Beyond this, you are on your own in terms of infrastructure if you seek to do a learning/optimization-based project. You are of course also welcome to implement or build off one of the earlier papers that does not require deep learning, and there are modern approaches that can be implemented with simple optimization that may not require extensive compute resources.
Please note that CSE 274 is a topics course, covering current
topics in computer graphics. Course content may change every year or
two years. You can see my website for previous iterations. This is
the first offering of the course focused on image-based rendering.
You are allowed to enrol and get full credit even if you have
taken CSE 274 on another occasion with me or other faculty.
Below are a few example images produced using some of the algorithms
and systems we will be discussing.
|
|
|
|
|
|
The course is targeted towards MS and PhD students with a knowledge of and interest in computer graphics (at the level of the introductory CSE 167 computer graphics course, or equivalent at another university). CSE 168 is helpful, especially to understand modern volumetric rendering for IBR, but not required (please consider signing up for CSE 168 on edX; if not I am offering it in spring quarter). We will cover some of the basis of physically-based Monte Carlo path tracing in this course for those who have not taken CSE 168, but that material is much more detailed and thorough. We encourage you to do the CSE 168 assignments until at least the basic path tracer (homework 3, and ideally homework 4) in a systematic fashion. Note however, that no prior knowledge of rendering or CSE 168 is required, and that the path tracing assignment is strictly optional/for your own benefit and not graded... you can also use an existing off-the-shelf black box path tracer (such as PBRT, Mitsuba or OptiX) for your project, as needed. However, if you have not written a path tracer or taken CSE 168 before, and you do end up building the path tracer, please include a link in your final project submission, to your full resolution grader feedback results. We welcome all PhD students working in graphics, vision, and robotics. We are willing to consider students who do not completely fulfill the pre-requisites in exceptional cases, if they have a strong interest in the material. Undergraduates who have taken CSE 167 (and/or other advanced courses such as CSE 168/169) are also most welcome, space permitting (space is not usually a problem).
The course is an integral part of the vision and graphics track for undergraduate, MS and PhD students (The instructor can help in getting any relevant permissions for it to count for credit towards the track as needed). It builds on the undergraduate graphics classes CSE 167 [taught by Profs. Chern/Li in the fall/winter] and CSE 168/169/190. If you like this course, you may also be interested in CSE 291s in winter and spring: machine learning for 3D geometry by Prof. Hao Su and spring courses on computational photography taught by Ben Ochoa, among others (note that the specifics of course offerings change from year to year, please consult latest update).
The course will consist of lectures on the relevant topics by the instructor, student presentations of papers covering current research in the area, and student projects. A syllabus/schedule is noted below. The grading will depend primarily on the final project. (Loosely, we note this as approximately 60% for the project, 30% for paper presentation and 10% for class participation, with the final project being the most critical aspect.) Students are expected to come to class regularly and participate in discussions, since this is an advanced graduate course.
In general, roughly (depends on the number of students in the course), 1 or 2 paper presentations will be required (if the number of students in the course exceeds the number of presentations, some may be done jointly by a group of two students). A project is not required for students taking the course S/U or P/NP; we leave it to your discretion if you want to register for 2 or 4 units in this case. An S/U registration is a good option for PhD students and others to read papers on this exciting topic and learn about the area without committing too much effort into a course project, and we encourage you to sign up. Auditors, who simply want to sit in on the course are also welcome; however, we prefer if you sign up for the course pass/fail instead [this just involves doing one or two paper presentations].
Students taking the course for a letter grade are required to do a project [this may be in groups of 2], give a presentation in class regarding their results, and also submit a final written report. Wide flexibility is available with respect to project topics, provided they relate loosely to the subject matter of the course. We expect that most projects will implement one or more of the algorithms or papers discussed in the course, showing results on some aspect of image-based rendering. We welcome suggestions from the students on alternative project ideas, and use your judgement (we do not want to constrain you in doing creative work that may build on some existing resources). The best projects will go beyond the published work in some way, such as trying out an alternative or better approach or trying to develop some variant or more general version of the technique. However, this is not essential; in general, students who fulfil all course requirements including a well-executed project will easily receive an A in the course.
As a potentially easier alternative to the project, we will also accept a well-written summary or tutorial, covering 3 or 4 papers. The best summaries will point out links between the papers not noticed by the original authors and suggest improvements or directions for future research. However, this option is recommended only as a last resort and will generally receive a slightly lower score; we prefer that you do a good project (which may involve understanding a few papers in any case).
For those seeking to use this course as a comprehensive for the MS degree there are a few restrictions. You must take the course for grades, you must do the project, the project needs to be done individually, and you must receive an overall B- or higher grade.
The lectures will be in EBU3B (CSE Building) 4140 from 9:30-10:50 on Tuesdays and Thursdays. Students are expected to come to the lectures and participate in discussion. Office hours will be immediately after class from 11-12 in the Professor's office, EBU3B 4118. You may also e-mail for another time to meet if that is not convenient. We also have a Piazza discussion board setup for this course if you are interested. It can be accessed at: https://piazza.com/ucsd/winter2026/cse274
We have a teaching assistant for the course, Nithin Raghavan. His e-mail is n2raghavan@ucsd.edu His office hours will be Mon and Thu at 3pm by the whiteboard outside CSE 4140. or you can e-mail him to schedule an appointment.