One factor that has drawn attention in recent years has been the arrangement of muscle scars on the shell. Large muscle attachment scars appear as raised annular elevations on the inside of the shell towards the rear end of the body chamber (in practice, they are more often observed in fossils as depressions on the internal mould). In the living nautilus, the muscles attached to these scars function in the retraction of the head (King & Evans 2019). Modern nautilus possess a pair of large lateral scars in an arrangement that has been labelled 'pleuromyarian'. However, many of the earliest cephalopods possessed a ring of numerous small scars, an arrangement referred to as 'oncomyarian'. Other cephalopods might have scars restricted to the dorsal ('dorsomyarian') or ventral ('ventromyarian') midline.
Another feature that has been called out has been the structure of the connecting rings around the siphuncle. Shelled cephalopods, you will recall, have the shell divided into chambers separated by septa. Though the bulk of the animal is found in the final body chamber, a fleshy cord called the siphuncle runs back through the remaining chambers. In life, the siphuncle is used to control the levels of fluid in the chambers, which in turn controls the animal's buoyancy. The boundary between the siphuncle and the surrounding chamber is marked a toughened sheath, referred to as the connecting ring. In the modern nautilus, the connecting ring is comprised of two layers, an outer calcareous layer and an inner chitinous layer. In comparable fossils, the latter chitinous layer has decomposed after death so only the outer layer is preserved. However, some extinct cephalopod groups preserve evidence of calcification in the inner as well as the outer layer. Based on the distinction between these two siphuncle types, Mutvei (2015) supported dividing most of the nautiloids between two major lineages, the Nautilosiphonata (with a nautilus-type siphuncle) and the Calciosiphonata (with the internally calcified connecting rings).
A couple of years earlier, the same author (Mutvei 2013) had proposed recognition of a superorder Multiceratoidea for nautiloids that combined multiple muscle scars with a nautilus-type siphuncle. Examples of nautiloid orders with such a combination included the Ellesmeroceratida (small nautiloids with densely placed septa), the Oncoceratida (often short, squat nautiloids) and the Discosorida (similarly squat forms with complex bulging connecting rings). All of these were found in the earlier part of the Palaeozoic with the oncoceratids dieing off in the early Carboniferous. Mutvei (2013) also included the coiled Tarphyceratida and the egg-shaped Ascoceratida in this group. Later, King & Evans (2019) redefined this grouping as the Multiceratia, excluding the Tarphyceratida and Ascoceratida on the grounds that they had ventromyarian rather than oncomyarian muscle scars. Mutvei (2013) suggested that, rather than representing retractor muscles, these smaller repeated scars were associated with an outgrowth of the mantle, either as tentacles or a muscular 'skirt', that was used to capture micro-plankton.
King & Evans (2019) proposed a reclassification of the subclass Nautiloidea between five subclasses defined primarily by muscle structure. Apart from the earliest oncomyarian Plectronoceratia, most 'nautiloids' could be divided between two lineages. On one side were the dorsomyarian Orthoceratia (usually thought to include the ancestors of the ammonoids and coleoids). On the other, the oncomyarian Multiceratia would eventually give rise to the ventromyarian Tarphyceratia which in turn included the ancestors of the pleuromyarian Nautilida. Note that many of the reocognised subclasses (and orders) remain paraphyletic but we are at least approaching a more informative picture of cephalopod evolution than the earlier unceremonious dumping into 'Nautiloidea' (I should probably also remind you that, for various reasons, most invertebrate palaeontologists still don't regard strict monophyly as a taxonomic requirement in and of itself).
The usage of muscle scars and connecting rings as classificatory keys is handicapped by the difficulty of observing them. As internal structures, they each require careful preparation of a specimen to observe. And once you've gotten to a position where you can see them, it seems not to be particularly easy to tell just what you're looking at. As a result, muscle scarring and siphon structure remains undescribed for the majority of nautiloid species. Judging the structure of connecting rings seems to be particularly challenging and some have gone so far as to suggest that purported different structures may be the result of post-mortem taphonomic processes (King & Evans 2019). Nevertheless, what we do know suggests that such features remain reasonably consistent within each of the well-recognised nautiloid orders. And Mutvei's (2015) concept of Calciosiphonata vs Nautilosiphonata does largely line up with King & Evans' (2019) dorsomyarian vs oncomyarian-ventromyarian lineages. There are, of course, some notable exceptions. Whether these will cause the developing structure to collapse, or whether they indicate mistakes in interpretation, only continued research will tell.
REFERENCES
King, A. H., & D. H. Evans. 2019. High-level classification of the nautiloid cephalopods: a proposal for the revision of the Treatise Part K. Swiss Journal of Palaeontology 138: 65–85.
Mutvei, H. 2013. Characterization of nautiloid orders Ellesmerocerida, Oncocerida, Tarphycerida, Discosorida and Ascocerida: new superorder Multiceratoidea. GFF 135 (2): 171–183.
Mutvei, H. 2015. Characterization of two new superorders Nautilosiphonata and Calciosiphonata and a new order Cyrtocerinida of the subclass Nautiloidea; siphuncular structure in the Ordovician nautiloid Bathmoceras (Cephalopoda). GFF 137 (3): 164–174.