Semi-spoiler:

4 is not enough
Example:
A knows B, doesn't know C or D
B knows A, doesn't know C or D
C knows D, doesn't know A or B
D knows C, doesn't know A or B