Neither the colour or/nor the size of the clothes suits/suit me.

Nor/suits

or/suits

and/suit

nor/suit

Generally, either is followed by or.

Neither is followed by nor.

Not only is followed by but also.

In the above sentence, the main subject of the sentence is size which is singular.