Explanation

“To” is used after following verbs if the object is a “person”.

Say, suggest, speak, reply, explain, complain, talk, listen, write, report, pray and describe”.

He said to me not to worry.