Fish in general are difficult to sex and distinguishing male from female tilapia can be a challenge. Tilapia grown in pond culture can have a problem with excess reproduction. This can lead to stunted growth and lower production rates.
To prevent this problem, farmers can use monosex culture by separating the males from females. Typically, males are preferrred because they grow to a larger size and have greater profit potential.
You can look for physical differences between the sexes. The following description is from Tilapia: Life History and Biology by Thomas Popma and Michael Masser:
“The sex of a 1-ounce (25-gram) tilapia fingerling can be determined by examining the genital papilla located immediately behind the anus (Fig. 1). In males the genital papilla has only one opening (the urinary pore of the ureter) through which both milt and urine pass. In females the eggs exit through a separate oviduct and only urine passes through the urinary pore. Placing a drop of dye (methylene blue or food coloring) on the genital region helps to highlight the papilla and its openings.”
The following picture gives you a reference for differentiating the sex of tilapia.
Home growers can use tilapia behavior to help sex their tilapia. Put a small number of tilapia in an aquarium along with some gravel. The male will typically dig a nest and defend it. Females will tend to hide unless they are spawning. You can remove males that display nesting behavior one by one and move them to separate containers. When the remaining fish no longer show nesting behavior, then you can assume they are all females. You can add one of the males back into the aquarium and start a breeding colony.