“Have you finished your lesson, Rani?” said Mr. Ali to his son.

Mr. Ali said to his son Rani that have you finished your lesson.

Mr. Ali asked his son Rani whether he had finished his lesson.

Mr. Ali inquired to his son Rani that whether he finished his lesson.

None of these

Mr. Ali asked his son Rani whether he had finished his lesson.

The sentence is a yes/no question, so "asked... whether" is correct.

"Have you finished" changes to "he had finished" in indirect speech.