High Performance Computing

This was my second course at OMSCS, and this was much harder than I could have anticipated. Firstly, taking this course in the condensed Summer semester is not recommended. This course is already extremely dense, and the assignments require a lot of time investment. Definitely take this course alone. This course is very theory-heavy, but for good reasons.

That being said, this was a very rewarding course, where discussions with other students greatly increased my understanding of the material. Despite the heavily theoretical material, the lectures are very well structured, and the TA's are very supportive.

There are 2 exams in this course, and they are both very difficult, however, both exams are curved. The course also had adjusted grade scales, where a C was >=64%, B >= 74.5%, and A >= 85%. Start the assignments as soon as they are available, as Spring and Fall semesters have 3 weeks between assignments, whereas Summer semester only has 2 weeks between assignments. There is also an extra credit assignment towards the end of the course, which in our case was worth an additional 10% of our grade.

This course was very challenging but rewarding. I came into the course with no HPC background and I didn't take Graduate Algorithms or Graduate Operating systems, but I did complete undergraduate algorithms and OS. This course teaches you how to be a better engineer and think differently. You will learn about how to write programs that work with different memory levels, shared memory and distributed memory. This exposes you to different frameworks like OpenMP, MPI, and CUDA. You get to develop on GaTech's PACE Cluster.

If these sound interesting to you, consider these warnings as well:

• Exams are difficult. The class average has around a 50 on each of the two exams but they are heavily curved. They consist of short answer questions where you write your own algorithm and analyze it
• The topics are niche. I found it very hard to find any resources outside of the lectures and readings to study the different sorting algorithms and algorithm analysis schemes (work and span models, alpha-beta models). If you are okay with reading papers and implementing them you should do fine
• You need to be okay with coding in C
• You will have to learn OpenMP, MPI, and CUDA with online resources or readings, there are no lectures for them
• Labs can be time intensive and PACE cluster can get crowded
• very low Ed Discussion engagement (at least this summer semester)

If you can get past those warnings, there's definitely a bright side. The instructors are very encouraging in getting you to learn. Each lab becomes less time consuming as the class progresses. On one lab, the grade thresholds were loosened after we submitted it. Each exam is curved and the class final grade itself has a curve. 85 was the cutoff for an A and 74.5 for a B this summer semester. If you are coming in with GA experience, you might be able to find this course less intense. I would not recommend taking this course in summer if you truly want to take your time and learn the topics. For me, I just wanted the difficult parts over with as fast as possible and this is an elective for the ML specialization. There were times I dreaded being in the course. My hopes are to apply HPC topics into distributed GPU training for AI models in the future. I would recommend the course but if you want to focus on AI, you might get away with just learning CUDA and PyTorch Distributed Parallel modules on your own time instead of taking this.

Very challenging course, as others have said. Start projects early, and don't leave your performance testing to the last minute. Project 1 particularly took me a long time to understand and complete, and had many students competing for the cluster near the due date. Exams were HARD. Since I was going for an A, I rewatched all lectures and attempted all of the provided example questions prior to the exams.

Having said that, the course is well run, and got more enjoyable near the end. TAs seem to care and are encouraging. There are weekly office hours and paper discussions, making this course feel a bit more alive and less like you are trying to figure this stuff out on your own.

Overall, I ended up with an A. I feel that if you truly put in the effort to succeed by doing well on projects (90+) and actually learn the material for the exams, you should do very well too. Just make sure you come in with a working knowledge of C (GIOS was good prep for me), and be very comfortable debugging your code.

Taking HPC felt like a real level up intellectually, but was also incredibly humbling. It's impossible not to be impressed by the genius behind some of these parallel and distributed computing algorithms, and it's scary to think about how clever the people who thought them up must be.

The labs were the most fun of any class I've done at OMSCS (with the possible exception of the RAIT projects). Chasing those performance scores had kind of a gamified feel, and whenever I scored full marks on one of them it was super satisfying. As I think some others have mentioned, the grading on these labs is not as finnicky as the GIOS projects, and it feels like you can concentrate on the meat of what you're supposed to be doing rather than spending endless hours testing for weird edge cases.

Time management on the labs was extremely important. When you get close to the due date, the PACE cluster gets jammed up with traffic, so you really need to get your work done several days ahead of time if you want a good score.

The tests were incredibly brutal, and despite having heard this ahead of time I was initially caught unprepared. Some of the questions on the midterm were so hard that I started laughing at them. This at least motivated me to try a lot harder in the second half of the class, and I was able to score close to the median on the final exam and then finish the class with an A due to high scores on the remaining projects and an ok score on the extra credit. So if something like this happens to you, and you have some solid programming skills, stick it out and you might do all right in the end.

Dr. Vuduc's lectures are excellent, though quite dense at times. I occasionally found myself rewinding the same 30 seconds of video half a dozen times to grok his points. The material is very different from a systems design class like GIOS, and requires more mathematical maturity and familiarity with analysis of computational complexity.

I remember when I started at OMSCS and I was looking through the course catalog, I figured that I would never in a million years try taking something like HPC. It just seemed too hard, and too different from the kinds of programming that I was comfortable with. I'm glad I had the guts to try HPC, as this class 100% lives up to it's reputation as one of the best and most challenging classes in the OMSCS program. I am also glad that I took the prerequisites seriously, as going into HPC without prior knowledge of algorithms and C programming would not have been a great idea.

It was my first course. I came with a non CS background, but decent programming work experience. I was not sure what I was getting into. I had a high level overview of parallel computing (OpenMP/MPI), but never wrote any useful code. Had no clue what CUDA is.

I was expecting the instructor would cover more of the MPI/OpenMP/CUDA details in the lectures, but those things are left to students to learn via the assignments, office hours, reading etc. The lectures mostly cover different general parallel algorithms for some standard problems, and teach how to analyse your parallel algorithms. In a way, this is great, because you would never learn this outside of a class.

Assignments were challenging. I found the labs 1-3 decrease in time required/complexity, which was a relief TBH. I spent insane amount of time on lab1. Grading was very generous. I scored below median in the mid-term, but I scored highest (I think) in the finals. I did put in a solid 20-30 hrs of prep for both mid term and finals. I found finals to be easier than mid term. Ended up with an A, which I wasn't expecting until almost end.

Lectures are super high quality and loved Prof. Vuduc's sense of humor! Keeps the lectures engaging.

I took HPC and found it to be a well-balanced yet challenging course. Lab 0 took only a few hours, while Labs 1-3 required a few days each to complete.

Lab 1 (involving OpenMP and CUDA) was the hardest in terms of maximizing performance for a 100% grade, while Labs 0 and 2-3 were more straightforward to achieve full marks.

The midterm and final exams were very challenging, even with open-internet and open-notes policies. However, they benefit from significant grade curves (+25%). I recommend practicing previous exam questions to prepare for the exams.

Compared to Compilers and Distributed Computing, HPC had significantly lighter project workloads, but the theoretical concepts were harder to grasp.

As a full-time student who has also taken GIOS, AI, Compilers, and DC, I spent about 10-15 hours per week on this course.

This was my last serious course in OMSCS, one more easy course and I'm done for the entire program. Some background: I have taken some challenging courses in through my time in this program (RL, GA, GIOS), I wouldn't call these super challenging, but did create some hiccups as I came from non-CS undergrad background (even though I code in my fulltime job). This has been the most challenging class for me, but I do like it a lot.

What I liked about it:

1. You really learn stuffs, useful stuffs, and will have a much better understanding of parallel programing and distributed system in general.
2. The project are rewarding and cements what you learned from lectures and textbook. You will realize that if you understand more, the better performance your code will be.
3. Even though there are private test cases for projects, but it's not too "edge-gy", meaning that you don't need to spend hours to figure out edge cases that are trivial. It is generally easy to get most of points from compiling and public/private tests, the meaty part is the performance part.
4. Textbook of this course and some reading materials are enjoyable to read (will be better if without any time limitation)
5. The TA team are super clear, project instructions are very clear as well.
6. Curve is generous, if you do your work, you get what you deserve.

What I didn't like:

1. Last project was graded on comparative performance, will be a time sink if you try to outperform your peers.
2. Exams, are pretty challenging and are pretty challenging to get a grasp on, 50/100 was like a average, but they are open book, open internet, so you don't have to memorize.
3. The last part of the course wasn't very well structured, it feels like crunching a bunch of things in a short period of time. I get it... it makes the learning curve not as steep for the first half of the course, but I did not enjoy the pace of last 3 weeks.

Some facts:

1. Unless you are a rock star, prepare many weekends doing nothing but projects. I had a regular fulltime job which is a little time demanding and I had to spend some weekends immersing in the projects.
2. The learning curve is very steep in the first half of the course, there are so much you need to learn before you can even do your projects: OpenMP, CUDA, MPI. Also, it is challenging to think in parallel programming's fashion, not to mention the algos with different types of analysis comparing to serial programs.

Says SP23 but was taken in SP24

This is a good course. Lectures are very good, albeit dense. A lot more math in this course than what I expected going in. The course readiness thing mentions calculus but you never use it (at least, not in lectures and not in our exams). Algebra and very basic linear algebra are good enough

Course is in 3 parts: memory-hierarchy-aware algorithms (e.g. external sorting), SMP multithreaded algorithms, and message passing distributed algorithms. Each section has its own mathematical model that's used for analyzing performance

Would recommend taking this course alone. Getting an A is already hard, but even if you manage to get an A there's still more to understand

This class was the halfway point for me, best one I've seen so far in OMS. Grateful for the professor and TAs

This was my 7th class in the OMSCS. It's got one of the best course material I've done, and I feel like I'm a much better engineer for it. The labs are hard but interesting and I definitely feel like I grok the concepts well afterwards.

Unfortunately the teaching staff were the worst I've had. I found them generally disdainful of the students, and my understanding of HPC was not aided by them. As an example, a student asked how a particular approach could possibly be efficient, given the overhead it requires. The response was "It's magic :-)", and no further elaboration.

As mentioned, the labs were difficult - I spent the entirety of several weekends on each one, and some performance tuning in the week when I could fit it in around my job. That being said, I was able to get good marks on them so it felt like the hard work paid off. There was also an extra credit project (which used to be a normal lab). The performance of your code is a significant part of the marks, which isn't surprising of a class called "High Performance Computing".

The midterm and exam were both very difficult. The midterm had a median of 46% and the final had a median of 45%. Both got flat grade adjustments. I didn't feel like they were good assessments of my understanding of the course material, and the instructors admitted that some questions were ambiguously expressed (which I read as "poorly written").

The course material has been shuffled around a bit since it was recorded, so I found there was some jumping around required to have the required background for the labs.

Overall, I'm glad I took this course as I feel like a better engineer now than at the start of the semester.

I'm surprised this course is rated as highly as it is.

I think the content that's attempted to be taught, parallel algorithms on distributed systems, is very rewarding but the video lectures do an extremely poor job of actually conveying the material if you don't at least have a bit of background in each specific topic. From the very first topic of "Intro to work-span model", I felt like a lot of introductory material was skipped and assumed student has the knowledge of. The worst part about this is, there isn't a proper textbook for the course that has the topics covered, and the TA's and instructor don't know of any external resources to refer you to learn the material on your own time. All in all, interesting material, very poorly taught.

Another huge con of this course is there is close to ZERO engagement in ed discussions. Nobody cares. The professor is not involved with the course at all, even though he's an active professor at Georgia Tech. The instructor is low energy and condescending and doesn't care at all. Some of the TA's are a bit more engaged, shout out to Hal Elrod, but overall the engagement is the worst that I've seen in the program by a long shot.

Assignments are good and rewarding, and not too time-consuming. Make sure you've figured out how to set up the virtual machine and you've got it working early on in the course.