I think you have too much time to sit and think, 'Am I the doer or not'. Just do what you are supposed to do. Every action has a reward; pleasant, unpleasant, or mixed. There are only three types of rewards. In every action there is something positive and there is something that is negative. An action cannot be 100% perfect. Focus on the positive things that develop out of an action. And don't worry about the effect of the action, just keep moving.

If you have done a wrong action, that has taught you a lesson not to repeat it next time. If you have done the right action, it encourages you to do many more of the same right actions.
Just move on, don't say, 'I am the doer', or, 'I am not the doer'. Anyway no one is a doer in one sense, and in another sense, yes, you are the doer, because inside of you there are two. There is a beautiful story in the Upanishads which says, 'On the tree of this universe, there are two birds sitting, one is enjoying the fruit, the other is just watching, like a witness'. So, there is one aspect in you that is doing everything. And there is one aspect of you that is not doing anything, that is much bigger. Both are present in you.

Now you might ask, 'How?' On one level, you find that you are changing. Your thoughts are changing, your emotions are changing, your body is changing. Yet at another deeper level, there is something in you that is not changing at all; it is the same. So it is not one or the other, it is both together simultaneously. When guilt eats you up, or pride takes over your head, then you should understand that you are not the doer. Do you get it?

People who have committed the biggest crime on the planet, somewhere deep inside, they think, 'Hey, I didn't do it, how did this happen'. They feel surprised and shocked, and they feel that they are not the doer, it just happened. If you talk to the biggest criminals on the planet, you will see that they find themselves being caught by an (criminal) act. Same way, someone who has done some excellent work, they also feel that it just happened; everything fell into place. That non-doership is an experience in itself.

When you have to do things, then don't say, 'Oh, I am not the doer'. Don't go into that mode. It will only tear you apart. If you have to focus on some work, focus and just do it.
I am not a doer, is an experience, sometimes it just dawns on you. And it is also a reminder to you when you are reeling in guilt, or in arrogance.