{Memory Bank}
This is powered, most of the research methods which are state-of-the-art techniques hinge on this idea for a memory bank. The memory bank is a nice way to get a large number of negatives without really increasing the sort of computing requirement. What you do is to store a feature vector per image in memory, and then you use that feature vector in your contrastive learning.
How it works:
How does the Memory Bank Work?
Motivation:
What you want is the features f and g to be similar. And you want features from any other unrelated image to basically be dissimilar.
In this case, what we can do now is if you want a lot of negatives, we would really want a lot of these negative images to be feed-forward at the same time, which really means that you need a very large batch size to be able to do this. Of course, a large batch size is not really good, if not possible, on a limited amount of GPU memory. The way to do that is to use something called a memory bank.
Memory Bank:
So what this memory bank does is that it stores a feature vector for each of the images in your data set, and when you’re doing contrastive learning rather than using feature vectors, say, from a different from a negative image or a different image in your batch, you can just retrieve these features from memory.
You can just retrieve features of any other unrelated image from the memory and you can just substitute that to perform contrastive learning. Simply dividing the objective into two parts, there was a contrasting term to bring the feature vector from the transformed image g(vI), similar to the representation that we have in the memory so mI. And similarly, we have a second contrastive term that tries to bring the feature f(vI) close to the feature representation that we have in memory. Essentially g is being pulled close to mI and f is being pulled close to mI. By transitivity, f and g are being pulled close to one another. And the reason for separating this out was that it stabilized training and we were not able to train without doing this. Basically, the training would not really converge. By separating this out into two forms, rather than doing direct contrastive learning between ff and gg, we were able to stabilize training and actually get it working.
References:
