The dictionary, with which we define the terms we use in language.
Okay, we could discuss dictionaries in general, their descriptive rather than prescriptive nature, their limitations, but we can skip that for the moment. Here's the Merriam Webster definition of gender.
https://www.merriam-webster.com/dictionary/gender
1)
a : a subclass within a grammatical class (such as noun, pronoun, adjective, or verb) of a language that is partly arbitrary but also partly based on distinguishable characteristics (such as shape, social rank, manner of existence, or sex) and that determines agreement with and selection of other words or grammatical forms
b : membership of a word or a grammatical form in such a subclass
c : an inflectional form (see inflection 3a) showing membership in such a subclass
2)
a : sex the feminine gender
b : the behavioral, cultural, or psychological traits typically associated with one sex
Clearly, we're not talking about definition #1a, b or c.
You seem to like definition #2a, but it seems clear to me that when trans people refer to their gender they are referring to something closer to 2b.
Now since you said the dictionary was that "with which we define the terms we use in language. " surely you must accept that 2b is a valid use of the word and that it does not in that usage refer to chromosomes or genitals.
While I'm at it, could you tell me which scientific discipline defines the word "table"?
If the meaning of table and what qualified as one were in hot dispute, I'd imagine that looking to fields that study tables would be quite useful.
Philosophers and sociologists and psychologists all study gender. If you insist that you're dictating a particular truth about gender, surely academic standards of terminology might come in handy.
I think determining things by observation is acceptable behaviour, yes. But the problem here is that, as usual with this topic, you're trying to mix morality with objective definitions. And also adding "insisting" to the sentence, which seems there to put a spin on the discussion.
We're talking about the behavior of calling someone by pronouns that they maintain are not desired or appropriate. Insisting seems like a very mild way of putting it, but if you insist

I'll say "firmly maintain" instead.
You're right we're looking at both moral and factual questions. I'll do my best to untangle them.
The question of whether it is right or appropriate to say something is always a moral/ethical question, regardless of whether or not what you say is factual. The factual nature of a statement can impact it's moral standing, it does not determine it.
So, to get back to the issue, let's start with the factual claim.
A trans man claims to have a masculine gender even though his sex is female.
The behavior we're investigating is firmly maintaining the practice of referring to this person as "she" and "her" despite this person's express communication of a preference to be thought of as a man and referred to by the pronouns "he" and "him".
Now correct me if I'm wrong, but your position seems to be that this person is incorrect in maintaining a masculine gender and the use of those pronouns, and because of that fact you feel morally/ethically justified in using "she" and "her" and referring to this person as a woman.
This person is clearly not referring to their DNA or genitalia when they define their gender, so one would not use definition of gender 1a to determine the truth or falsity of this person's statements. The truth or falsity would rest somewhere within behavior, culture or psychology. Now an academic definition would give us more specificity, but since we're not using those, and the dictionary has defined our term, the truth rests on there simply being some element within those three domains that is masculine.
Your previous standard of determining the truth value of their statement seems flawed from this perspective. Not being a psychologist or privy to a detailed psychological definition of gender, it seems from an epistemological point of view that one ought to rely on this person's self report of their psychological state. They have far more access to both the information and the standard than you and no particular reason for dishonesty. While delusion is possible, I suppose, we'd need to more specifically define what psychological state was misperceived.
Now back to the moral/ethical part of it. Engaging in a behavior with no practical workplace value, targeted toward a particular coworker, repeatedly over their objections is hard to justify.
Peeking in someone's cubical and mooing at them, even after they repeatedly made clear it was very unpleasant would be at a minimum, a jerk move. Now if your action directly irritated a reasonable sensitivity, for instance, if you poked your head in an overweight coworker's cubical and yelled "you're fat" on a daily basis, after they made it clear that it hurt them, that would be harassment. Even though the statement "you're fat" is true, it's still a major jerk move.
The behavior we're discussing more broadly is such a jerk move, but to add to that, it isn't even true. The coworker described earlier is not objectively of a feminine gender. So you would be both ethically and factually wrong to insist on referring to him as such.