“I was afraid of the man that held a gun in his hand.”
Because “man” refers to a person, it should read: “I was afraid of the man who held a gun in his hand.” (Actually, that could be shortened to read “I was afraid of the man with a gun.”
“Who,” “which,” and “whose” refer to people; “that” refers to objects. They are all relative pronouns that introduce a subordinate clause.
The tricky part is when you’re using the words “group” or “couple” or “crowd.” You are referring to people, but the words are themselves objects. Technically, you should use “that” afterwards, but there are times when “who” really does sound more appropriate.
Use your judgment.