It’s mostly the “hobby” bit, which implies that this is one of your favorite, regular free-time activities. If you are someone who smokes weed a few times a year in social situations, that is a different situation.

A lot of people would find it difficult to be around someone who spends their time smoking unless they are really into it too. The smell especially can be really grating/annoying, and the person smoking often doesn’t realize because they have become smell-blind over time (similar to tobacco smokers).