But when does anyone actually use a one-hot encoding as a vector space? They are used as a lookup table into a different representation, and the one-hit encoding is there to simplify mathematical description, not to be used as a vector space. Or have you experienced systems where people use one-hit encodings?
While I would agree that technically, in terms of raw dimensionality, a one-hit encoding is larger dimension than the embedding, that would be a "lower" dimensionality, not "low" dimensionality. Who uses "low" as an absolute term when referring to 300-2000 dimensions?
While I would agree that technically, in terms of raw dimensionality, a one-hit encoding is larger dimension than the embedding, that would be a "lower" dimensionality, not "low" dimensionality. Who uses "low" as an absolute term when referring to 300-2000 dimensions?