Thank you. If we consider it as a supervised (not zero-shot) problem which is mapping input image to target class, your proposed solution probably would work. How accurate? — I don’t know, you should give it a try? :D But remember that there are already several deep convolutional networks which can perform this task easily and incredibly accurate. They just use image embeddings. The main motivation that we use class embeddings is to be able to overcome the zero-shot problem. I don’t think that one can come up with a solution just by using regression. A more sophisticated solution is needed.

Best.