[Re] A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space
Nov 28, 2022
Speakers
Benjamin Džubur
Speaker · 0 followers
About
Scope of Reproducibility The authors of the paper, which we reproduced, introduce a method that is claimed to improve the isotropy (a measure of uniformity) of the space of Contextual Word Representations (CWRs), outputted by models such as BERT or GPT-2. As a result, the method would mitigate the problem of very high correlation between arbitrary embeddings of such models. Additionally, the method is claimed to remove some syntactic information embedded in CWRs, resulting in better performance on semantic NLP tasks. To verify these claims, we reproduce all experiments described in the paper. Methodology We used the authors' Python implementation of the proposed cluster-based method, which we verified against our own implementation based on the description in the paper. We re-implemented the global method based on the paper from Mu and Viswanath, which the cluster-based method was primarily compared with. Additionally, we re-implemented all of the experiments based on descriptions in the paper and our communication with the authors. Results We found that the cluster-based method does indeed consistently noticeably increase the isotropy of a set of CWRs over the global method. However, when it comes to semantic tasks, we found that the cluster-based method performs better than the global method in some and worse in other tasks, or that the improvements are within margin of error. Additionally, the results of one side experiment, which analyzes the structural information of CWRs, also contradict the authors' findings for the GPT-2 model. What was easy The described methods were easy to understand and implement, as they rely on PCA and K-Means clustering. What was difficult There were many ambiguities in the paper: which splits of data were used, the procedures of the experiments were not described in detail, some hyperparameters values were not disclosed. Additionally, running the approach on big datasets was too computationally expensive. There was an unhandled edge case in the authors' code, causing the method to fail in rare cases. Some results had to be submitted online, where there is a monthly limit of submissions, causing delays. Communication with original authors We exchanged many e-mails with the authors, which were very responsive and helpful in describing the missing information required for reproduction. In the end, we still could not completely identify the sources of some remaining discrepancies in the results, even after ensuring the data, preprocessing and some other implementation details were the same.Scope of Reproducibility The authors of the paper, which we reproduced, introduce a method that is claimed to improve the isotropy (a measure of uniformity) of the space of Contextual Word Representations (CWRs), outputted by models such as BERT or GPT-2. As a result, the method would mitigate the problem of very high correlation between arbitrary embeddings of such models. Additionally, the method is claimed to remove some syntactic information embedded in CWRs, resulting in better performance…
Organizer
NeurIPS 2022
Account · 956 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization
Watch later
Hai-Ming Xu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Contextual Bandits with Knapsacks for a Conversion Model
Watch later
Zhen Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Welcome
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
Watch later
Sanae Lotfi, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Semantic Supervision for few-shot generalization and personalization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Fair and Efficient Allocations Without Obvious Manipulations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%