The distinguishing feature of the two subgenera was that Antiplanes had sinistral coiling, while Rectiplanes had dextral coiling. The two terms refer to the direction in which the shell coils (its chirality). If you hold a gastropod shell with the tip pointed upwards and the opening at the bottom facing towards you, then in a dextral shell the opening will be on the right-hand side while a sinistral shell will have the opening on the left. The two forms of coiling are mirror images of each other. This has more than merely aesthetic consequences for the individual snails - because snail anatomy is asymmetrical, these differences in orientation are also reflected in the position of the genital openings. Snails with different orientations have the genital openings on different sides of the head, and in species that mate face-to-face the genital openings cannot be brought into contact between snails of different orientations. Usually the orientation of the body matches that of the shell, but there are some gastropods in which the two are opposite - as pointed out by Gould et al. (1985), this condition (called hyperstrophy) is actually a kind of developmental illusion caused by a reversal in direction of growth between the protoconch (the larval shell) and the teleoconch (the adult shell).
Dextral coiling is far more common in gastropods than sinistral coiling. Gould et al. (1985) noted that from hundreds of thousands of examined specimens of the West Indian snail Cerion, only six sinistral examples had been recorded*. When Powell (1966) compiled his catalogue of turrids, Antiplanes contained eleven dextral species compared to four sinistral species. The reasons for this imbalance remain entirely unknown. In many cases that have been investigated, a single gene appears to be involved in determining gastropod chirality. Differences between dextral and sinistral forms other than mere chirality are generally minor - Gould et al. (1985) did note morphometric differences in e.g. size of the aperture between sinistral and dextral Cerion, but Dietl & Hendricks (2006) found that sinistral individuals of Planaxis were more resistant to predation by crabs than dextral individuals. Some of the few clades of predominantly sinistral taxa have been perfectly successful, thank you very much, while a few gastropod taxa even maintain populations that are stably polymorphic for shell chirality.
*In the typical Stephen Jay Gould style that people either love or loathe, the paper opens with, "Vishnu, the great Hindu god of preservation, holds in one of his several hands a shell of the genus Turbinella".
To get back to Rectiplanes, the obvious question is whether chirality can be regarded as a significant factor in taxonomy. Does it indicate a close exclusive relationship between members of one or the other subgenus, or has chirality changed back and forth multiple times in the history of the genus? Unfortunately, no-one seems to have looked at the interspecific phylogeny of Antiplanes itself, but the general indication from other taxa seems to be that chirality alone is not a reliable indicator of phylogeny. Because only a single gene is generally involved in determining chirality*, mutations might be expected to happen fairly recently, and even if selected against somehow, recessive mutations might still persist in the population via heterozygote carriers. What is more, chirality seems to be determined not by the genotype of the individual, but the genotype of its mother (Gittenberger, 1988; Anonymous, 2005). In other words, a genetically sinistral snail born of a dextral mother will be phenetically dextral, but all of its offspring (whether genetically sinistral or heterozygotic) will be born sinistral. This means that even though sinistral and dextral individuals may be unable to mate directly, gene flow can still occur between sinistral and dextral morphs in the population. It also means that rather than appearing as isolated individuals doomed to pass away without finding a suitable mate, reversed-chirality individuals are more likely to appear in numbers, allowing for a greater chance of establishment.
*A big caveat needs to be slapped across what I'm saying here. Probably because of matters of pragmatism, genetic determination of chirality has been almost exclusively studied in members of the Pulmonata, the clade of gastropods that includes the vast majority of terrestrial snails and a fair proportion of freshwater ones. Therefore, caution should probably be exercised in extrapolating what we know about chirality in Pulmonata to gastropods that lie outside Pulmonata - such as Antiplanes.
Such factors mean that we might reasonably expect reversals in chirality to occur multiple times within a clade that includes examples of both chiralities, and that monophyly of one or the other chirality cannot be assumed. For instance, Ueshima & Asami (2003) found that while sinistral species of the land snail genus Euhadra had derived from a single ancestor, reversals to dextrality had occurred multiple times within the sinistral clade. Indeed, the dextral E. aomoriensis had potentially arisen polyphyletically from its sinistral ancestor E. quaesita. Therefore, there is little justification in maintaining Antiplanes and Rectiplanes as separate taxa without anything to separate them other than chirality.
Anonymous. 2005. Speciation begins, but doesn't end, with the twist of a shell. PLoS Biology 3 (9): e330.
Dietl, G. P., & J. R. Hendricks. 2006. Crab scars reveal survival advantage of left-handed snails. Biology Letters 2 (3): 439-442.
Gittenberger, E. 1988. Sympatric speciation in snails; a largely neglected model. Evolution 42 (4): 826-828.
Gould, S. J., N. D. Young & B. Kasson. 1985. The consequences of being different: sinistral coiling in Cerion. Evolution 39 (6): 1364-1379.
Powell, A. W. B. 1966. The molluscan families Speightiidae and Turridae: An evaluation of the vaid taxa, both recent and fossil, with lists of characteristic species. Bulletin of the Auckland Institute and Museum 5: 1-184.
Ueshima, R., & T. Asami. 2003. Single-gene speciation by left-right reversal. Nature 425 (6959): 679.