Ok. Your question is very pertinent.
Many aspects should be taken into account when working with the concept of “rarity”.
As I explain in my article, many mineralogists have described rarity (Weiß (2018), Hazen and Ausubel (2016) among others) including factors such as formation conditions (as you point out), sporadic character, difficult access, strange composition, scarcity of deposits... (I develop all this aspects).
So, the definition of "rarity" must always be related to the bases you use to express/calculate… it.
When I define the rarity index (iR), I focus it only on the chemical composition of the mineral, that is: the elements and their abundance in the earth's crust, and the number of atoms of each element in the formula. And always working on a logarithmic basis so that very small (or high) abundance values can be smoothed out to avoid unmanageable data (iR moves from 2 to 12).
This method allows obtaining a quantitative value for rarity, not subjected to qualitative issues.
There are many more arguments in my description but this is a rapid summary of it.
I strongly emphasize the fact that this is one way to determine rarity, but that the other existing ones are also valid, of course.
BR
Weiss, Stefan (2018): Das Grosse Lapis Mineralienverzeichnis, 7a ed. Munic: Christian Weise Verlag GmbH
Hazen, R.M. and Ausubel, J.H. (2016): “On the nature and significance of rarity in mineralogy”, American Mineralo¬gist; vol. 101, pp. 1245-1251.