How do people know you’re not a native speaker?
You mix up “to me” and “for me”!
Yes, it can be confusing because there’s some situations where both can work.
I try to avoid explaining grammar as much as I can, but in this case, it really effects how you sound. Let’s break it down simply so you can improve your English fluency.
To Me
“To Me” is focused on direction. For example:
Those are all pretty easy to grasp. You probably know how to use “to me” in this way. Now let’s discuss “for me”.
For Me
When you use “for me”, you’re focusing on “purpose/reason”.



If you get a good grasp of those differences, your speaking will sound much better.
Now you know we would never say “Can you work to me?” because this context has no focus on direction. The focus would be on purpose or reason. You would ask someone to work for you because 1) you need more employees 2) you don’t feel well 3) you need time off for a vacation. Those all have a purpose.
Remember, mixing up “to me” and “for me” aren’t severe enough to confuse other people you’re talking to. They’ll still be able to understand what you mean. And that’s probably why your American friends never correct you on this mistake.