<aside> 👉 The Neural Explanation project aims to understand what deep neural networks learn during the training process by analyzing what individual neurons are able to recognize in terms of concepts (e.g., cat, building, etc). Specifically, we aim to achieve this goal by associating logical rules to each neuron that express the context in which those concepts are recognized (e.g., (Cat OR Dog) AND NOT person)). The rules are usually extracted by combining search algorithms, clustering algorithms and statistical analysis of neurons activation.

</aside>

Fall 2025 Auditor Project

The Fall 2025 Auditor Project for the neuron explanations project will be to compute, analyze, and compare explanations for neurons in convolutional neural networks across different layers of one’s choice.

Week 1: AIEA Lab Onboarding

Week 2: Project Onboarding (Get the reference code repository running locally. )

• Neurons Choice

Week 3: Nautilus

Week 4: Get the reference code repository running on Nautilus in a headless environment (you will need to be well acquainted with the terminal). Document the process.

• Note that this task is mandatory if your workstation cannot run the “standard” configuration, as completing the next steps will not be possible otherwise.

Week 5: Read the paper and write a small report summarizing it in your own words.

Week 6: Visualize and comment the explanations for the selected neurons of the last layer in ResNet18.

• Choose another layer in the network and “reserve/claim” it with the teaching team.

Week 7: Visualize and comment the explanations for 50 neurons of the chosen layer in ResNet18 (or all the neurons if the layer has fewer than 50 neurons). Use the same seed as in the previous step. What went well? What didn’t? What are the differences in explanations?

Week 8: Test the quality of the explanations computed at the previous steps and comment the results

Week 9: Modify the clustering algorithm (i.e., by changing its parameters or the algorithm itself) with the goal of finding a configuration that significantly changes the explanations and comment on the explanation robustness.

Week 10: Auditor Offboarding + final report writing (5 pages max)

(CMPM 118 Only) Policy on Bonus Point Replacements: You can use bonus points to replace a maximum of two deliverables of this project. You may use bonus points for replacing Week 4, Week 5, or only one between Week 7 and Week 9

• For example, missing deliverables 4 and 9 and collecting at least 2 bonus points to compensate for them is fine. Conversely, if you miss both the 7 and 9 deliverables, you cannot complete the course even if you have more than 2 bonus points.