The problem fundamentally is yes it's your data. It's what makes you, you. The reason this is important is because if you can't control this data it can be used against you. At some point in the near future if not already you'll have job applications rejected based on third party data you didn't sign up for and can't opt out of. It will impact your healthcare, your loans, what prices you see when you shop and what items are shown. In essence what makes you, you is used solely for controlling you and the options available to you.
The issue then isn't the data, but the fact that it could be used in those ways. Because if it's legal for companies to use that data (even if it's owned/controlled by the user), they will incentivize people to share it in order to give them better prices/service, and the end result will be the same.
