So the short answer is that Dean has shown himself to be willing to resort to physical violence to get his point across to Sam. He doesn’t do it all of the time; he doesn’t even have to do it often. He does it often enough that his willingness to employ that method is know to the audience. “Pilot,” “Bloodlust,” “Metamorphosis,” “You Can’t Handle The Truth” and “The Girl Next Door” are examples of times when Dean has lashed out physically at Sam.
The longer and more complicated answer (you knew there would be one) is that we see enough of the physical side of the abuse to know that it is a learned behavior. We see more of it in the early seasons. I’ve seen some people justify it as just normal brotherly interaction, because men of that generation are naturally physical with each other… And very often men of that generation - my generation, I’d add - will slug each other to express irritation. I’ve seen it happen but I want to point something out and that’s two words: each other. Sam doesn’t defend himself when Dean lashes out physically. He’s learned not to fight back.
You know who else Sam doesn’t fight back against? John Winchester, when John manhandles him in Dead Man’s Blood. Dean is a staunch defender of John’s parenting style toward Sam from the pilot on. Dean learned, at the same time that he was learning that Sam was his responsibility, that physical “correction” was one of the ways he was taught to keep Sam “in line.”
So I honestly think that it’s something that Dean learned as a child, both because it’s something that was done to him as a disciplinary tool and because he wasn’t taught that it wasn’t acceptable to express his anger and frustration that way. Kids will naturally lash out, it’s what they do, and if the parents or caregivers don’t teach them that it’s wrong they don’t learn that it’s not okay to hit.
Now again, I’m not saying that Dean sits there and looks for opportunities to beat the snot out of Sam on a daily basis. That would be a caricature. He is physically abusive, but physical abuse doesn’t mean that the person on the receiving end is covered in bruises and black eyes on a daily basis. The threat of abuse, the knowledge that the abuser is willing to use physical violence, is sufficient and those factors exist between Dean and Sam.
There is a tendency to see physical abuse as “real” abuse and psychological abuse as not really abuse. The OP frames the question this way. Some psychologists will say that psychological abuse is actually harder to overcome than physical abuse. After all, bruises, broken bones and an actual physical act are very concrete. The victim can point to observable injuries that validate the fact that they are being abused. Plus, although the fear and knowledge that they can be hurt at any time is always part of physical abuse, the bruises fade, broken bones heal, our minds tend to forget physical pain (ask any woman who has been in labor, what it felt like fades).
Psychological abuse stays with a person. Hearing over and over that you don’t love your family, you don’t care about saving people, you are weak because you got addicted, you are gullible and stupid because you got played, you can not be trusted because you can’t eventually works its way into a person’s mind and self image. If, as in the case of Dean, the abuser also controls the view the outside world has of the victim, the world starts reinforcing the idea that the view of the abuser is right and the victim “deserves” to be told about their many, many failings and that these failings are absolute truth. Bobby is a great example of this. Bobby did not even bring up the issue that Sam could die during detox until it appeared that Sam was actually dying. Then he bought Dean’s belief that Sam deserved to die because being an addict meant Sam wasn’t human. Later in season five, Bobby was stunned to see Sam saving the victims of Pestilence in the plant that was manufacturing the Croatoan virus. He was stunned even though he admitted that Sam had been running into burning buildings since he was 12. Even though he had worked with Sam in season three before the demon blood addiction. Bobby was able to hold two different versions of Sam in his head and to decide that DEAN’S version of a Sam who is unable to resist evil or do good was right. This continues to some extent into season six where Bobby rejects Sam after his soul has been returned. Yes, Soulless!Sam had tried to kill him, but Bobby still mistrusted Sam with his soul, in part because Bobby sees Sam as not good enough. After all, Bobby accepted and protected Karen, who had also tried to kill him and who was definitely a zombie, something he knew was dangerous and lethal. But Bobby believed she was good at her core. His treatment of Sam indicates that he did not believe that about Sam, even though his personal experiences showed him Sam was.
Basically, psychological abusers can make the world complicit in their abuse. They define who their victim is and what they are capable of. They convince the victim. This makes even recognizing that the victim IS a victim hard. After all the abuser is really just a put upon martyr who is doing their best to take care of and love a basically flawed and unlovable partner. Dean is just a martyr who is doing his best to take care of Sam, who was chosen by evil, succumbed to evil, never loved Dean enough and never appreciated that Dean gave up his LIFE to be with unworthy, unlovable, weird Sam. And Dean did it all while being abused by his father. Dean stood between Sam and John’s neglect and abuse even though we have seen time and time again that Dean DIDN’T. But Dean’s is framing the story. Sam has been cut off from outside contact and validation. Sam has been told “no chick flick moments” and Dean does not permit Sam’s emotions to be told to Dean. Dean is in control of the situation and Sam only has himself to rely on to see when he is being abused and Sam has bought into Dean’s view of Sam. It took being possessed against his will by an angel (like physical abuse and observable, concrete act that can’t be denied or explained away) to have Sam fight back against Dean’s control and abuse. It is why he still has trouble stating what was wrong about the possession, because Dean is still framing the argument.
Abuse is a complicated, difficult subject, but saying that it’s “harsh” to call Dean an abuser because the abuse he deals out is only psychological and emotional is misrepresenting abuse. ANY kind of abuse is abuse and anyone who abuses someone gets to be called an abuser