It’s been years since the terrible claims first surfaced, but Sean Penn insists that he never physically attacked his ex-wife Madonna with a baseball bat. The pair had a turbulent relationship in the 1980s, with outlandish allegations accusing Penn of tying Madge to a chair one day and striking her with a bat the next. Penn, the star of I Am Sam, recently spoke with the New York Times about his prior relationship with Madonna, and he continues to deny the most serious allegations against him.

It should be noted at this point that Madonna has denied that Penn was ever physically violent to her, even testifying in court on his defense in a defamation action over the allegations. However, some argue that there is rarely smoke without fire, and conjecture has surrounded the now-63-year-old Penn for decades. In a recent interview with the New York Times, Penn admitted that one of his and Madonna’s disputes resulted in a SWAT team being sent to their home, but that this was not due to his physical violence.

Penn described in his interview how his ex-wife called the police because she was concerned about the presence of guns in the home. That appeared to be enough to set the law in motion, as a SWAT squad stormed into the couple’s home shortly after, smashing windows to gain entry before handcuffing Penn. “I had a freaking SWAT team come into my house,” Sean told The New York Times.

“I said, ‘I’m not coming out. I am going to finish my breakfast. The next thing I knew, windows were being broken all around the house, and they had entered.Penn and Madonna divorced in 1987, after two years of marriage. Certain allegations began to circulate, implying that Penn had been physically aggressive. The actor described how he heard gossip about having “trussed her up like a turkey,” however he said, “I didn’t know what ‘trussed up’ meant, first.”

Penn went outside to smoke one morning while with a new lover and was later challenged about beating Madonna in the head with a baseball bat. “I didn’t know what the hell she was talking about,” he remembered. “Now, I believe it is fair to state that I am not the world’s biggest guy. But if I hit Mike Tyson in the head with a baseball bat, he will go to the hospital.” In the interview, the actor also stated that he and his ex-wife are on good terms. “It turns out it’s a lot quicker to repair a friendship after divorce if there are not kids involved,” he told me.