Joseph was left out of the two-match home series after Cricket West Indies (CWI) announced he was unavailable due to "personal reasons." However, Sammy later said the fast bowler had "declined selection," prompting the pacer to respond publicly.
Speaking on the Good Morning Jojo Sports Show, Joseph said Sammy's comments were unnecessary and misleading.
"We had a chat, myself and the head coach. I've also had a chat with CWI and if you realise the statement CWI released said personal reasons. Mr Sammy, being around for a long time, he knows exactly what that statement would have done so, to be honest, I have nothing to say to Mr Sammy in terms of his statement.
"Like I said, he knows exactly what that statement would have done and the narrative it would have pushed in the public."
Joseph explained that his decision was based on workload management following a lengthy injury layoff. The fast bowler missed nearly a year of international cricket after suffering a second stress fracture in his lower back and only recently returned during the Test series against Sri Lanka.
He said playing two Tests against Sri Lanka followed immediately by five ODIs against New Zealand left him feeling physically unprepared for another Test series.
"Obviously, I've been out for an entire year. I haven't played at international intensity for so long after having a second stress fracture.
"We spoke about me playing the two Test matches here, playing five back-to-back ODIs and how my body was feeling and I knew to myself, my body was not ready to go into another Test series. So, the statement in itself is controversial, without context."
Despite the public disagreement, Joseph insisted it would not affect his commitment to the West Indies team.
"I'm a professional, so I'm going to go to work. I am representing the West Indies. I'm not representing Daren Sammy, so for me, I'm neither here nor there in terms of a relationship.
"I don't go there to make friends. I go there to work and do a job and represent the people of the West Indies."
West Indies currently lead the two-match Test series against Pakistan 1-0, with the second Test scheduled to begin on Sunday. Their next international assignment is a white-ball tour of India in September-October.