It is a widely-known secret that the fossil record is heavily biased towards hard parts of organisms. Soft body parts generally rot away before they can be fossilised, and usually only a shell, a bone, a piece of wood or something else reasonably crunchy has a chance of lasting long enough to be preserved. This is why sites that do preserve soft body parts, such as the Burgess Shale, Mazon Creek or Liaoning, cause such a sensation and are so significant, because the remains they present us with are so rare.
But for the vast majority of cases, we must make do with the occassions when the fossil record is willing to throw us a bone (ha bloody ha). And while palaeontologists are sometimes able to claw a simply amazing amount of detail out of just the hard parts of an organism, sometimes the information available is frustratingly incomplete. What can you say if all you have is a tube?
Take a look at the pictures at the top of the post. They look to show pretty similar organisms - in fact, if you know how to tell one from the other, you're more than a few steps ahead of me. Yet you've probably guessed already that they don't*. These are not examples of the same family - they don't even belong to the same phylum. The shells on the left (from here) belong to molluscs (gastropods) of the family Vermetidae, while those on the right (from here) are annelid worms of the family Serpulidae. Both have adopted a fairly simple tubular habit, with little in the way of extravagant ornamentation.
*After all, why would I have brought in all the rhetoric if they did?
Were you to find a fossil example of either one of these, however, all would not be lost. Molluscs and tube-worms lay down their shells in different ways, so if you knew what to look for you could tell them apart. Once you had identified your fossil as one of the above, then you could infer a lot more about what the soft parts of the animal may have looked like. Both Vermetidae and Serpulidae are around today, and the soft anatomy of the living species has been well-studied. But there are other tubular shells in the fossil record that don't have modern representatives. One might be tempted (and many have done so in the past) to compare them with modern tubular shells in molluscs and/or annelids. But mineralised skeletal structures have evolved independently at least in foraminiferans, cnidarians (multiple times), annelids, molluscs, bryozoans, brachiopods, echinoderms and vertebrates - it is quite believable that they may have evolved in other clades as well. So for now, most of these tubular fossils get relegated to the howling wasteland of incertae sedis (Malinky et al., 2004).
Hyolitha: Hyoliths (the name means "tongue stones") are conical shells found from the Early Cambrian to the Mid Permian. The illustration (from the Smithsonian) shows Haplophrentis, a member of the hyolith order Hyolithida, which possessed a ventral projecting ligula and two projecting side-arms called helens (structures unique to hyolithids that, in the absence of an appropriate descriptive term, Charles Doolittle Walcott apparently named after his daughter). Members of the other order, Orthothecida, lack these structures. Both orders have an operculum closing off the main shell - retractable in Orthothecida but external in Hyolithida. The function of the helens is rather uncertain - muscle scars close to them suggest that they were quite mobile and might have been used to move the animal across the sea floor (Mus & Bergström, 2005), but this seems in contradiction to their delicate structure. They may have been used to hold the animal upright on soft sea-bed. They have also been suggested as supports for an external tissue system for feeding, respiration or other nefarious purposes.
As for the affinities of hyoliths, most authors have associated them with molluscs, due to similarities in shell structure and composition. One genus of hyolith, Gompholites, preserves serial muscle scars that might indicate a segmented structure that would be inconsistent with molluscan affinities (though at least some molluscs possess/ed serial structures - viz. Neopilinida and Acaenoplax), but other hyoliths do not show such an arrangement and the features seen in Gompholites are generally interpreted as representing successive scars left by chances in the muscle attachment site as the animal grew (Mus & Bergström, 2005). Some remains of hyoliths show signs of a looped gut similar to that of the modern Sipuncula (peanut worms), and some authors have suggested a relationship of hyoliths to the latter (Runnegar et al., 1975). However, more basal fossil sipunculans possess a straighter gut (Huang et al., 2004).
Tentaculitoidea (Cricoconarida): Tentaculitoids are known from possibly the Ordovician (Malinky et al., 2004 - the Ordovician fossils are not definitely tentaculitoids) to the earliest Permian (Niko, 2000 - image from here). The type genus, Tentaculites, was originally identified (somewhat presciently - see later) as spines of brachiopods, and the name refers to the belief that they were appendages of crinoids (Schlotheim, 1820). Tentaculitoids are narrow conical fossils that are radially symmetrically along the long axis. There are two major orders - the Tentaculitida had heavier shells and were probably benthic, while the thinner-shelled Dacryoconarida may have been planktonic (there are also a number of smaller orders that have been counted as tentaculitoids, but authors have differed on these). Tentaculites has an annulated structure (as shown in the photo), and once it was recognised as an independent animal it was interpreted as an annelid due to these. Other tentaculitoids do not all show these annulations, and authors have also suggested a mollusc affinity (Yochelson's  review of a book on tentaculitoids is extremely telling in its reference to the "molluscan-annelid question", seemingly overlooking that there might have been other alternatives). The microstructure of tentaculitoid shells is very different from molluscs, and a similarity and potential affinity with Brachiozoa has been suggested (Herringshaw et al., 2007), though doubt has apparently been cast on whether a phoronid-style lophophorate feeding system is compatible with a planktonic life-style.
Cloudina: Cloudina has the distinction of being one of the earliest shelled animals to have ever existed, dating from the Ediacaran (drawing from Palaeos). It is constructed from a series of internested cup-shaped tubes. The affinities of Cloudina are completely obscure. In a seemingly not-yet-published manuscript available online, Miller compares Cloudina to annelid tubes, but with quite a bit of uncertainty.
Hyolithelminthes: Hyolithelminthes are elongate phosphatic tubes from the early Cambrian (image from Clausen & Álvaro, 2006). They are part of a sizable collection of phosphatic taxa from that time, though over time these forms mostly became extinct and were replaced by taxa with carbonate skeletons - today, relatively few taxa (such as linguloid brachiopods) possess skeletons of calcium phosphate. For a long time it was believed that fossils such as Mobergella represented opercula of hyolithelminthes, but these are now regarded as independent animals (Bengtson, 1992). The affinities of hyolithelminthes are unknown - a recent paper apparently aligns at least one hyolithelminth genus with cnidarians, but I haven't read the paper in question (Vinn, 2006).
Though this has turned into something of a major post, I could still cover many more examples of tubular problematica. Cornulitids, sphenothallids, paiutiids - the list goes on and on. But in the interests of sanity (and not wasting my entire weekend), I'll get out while I still have my dignity.
Bengtson, S. 1992. Proterozoic and earliest Cambrian skeletal metazoans. In The Proterozoic Biosphere: A multidisciplinary study (J. W. Schopf & C. Klein, eds.) Cambridge University Press.
Clausen, S., & J. J. Álvaro. 2006. Skeletonized microfossils from the Lower–Middle Cambrian transition of the Cantabrian Mountains, northern Spain. Acta Palaeontologica Polonica 51 (2): 223-238.
Malinky, J. M., M. A. Wilson, L. E. Holmer & H. Lardeux. 2004. Tube-shaped incertae sedis. In The Great Ordovician Biodiversification Event (B. D. Webby, F. Paris, M. L. Droser & I. G. Percival, eds.) pp. 214-222. Columbia University Press.
Niko, S. 2000. Youngest record of tentaculitoids: Hidagaienites new genus from near the Carboniferous-Permian boundary in central Japan. Journal of Paleontology 74 (3): 381-385.
Runnegar, B., J. Pojeta, N. J. Morris, J. D. Taylor, M. E. Taylor, & G. McClung. 1975. Biology of the Hyolitha. Lethaia 8: 181–191.
Schlotheim, E. F. von 1820. Die Petrefactenkunde auf ihrem jetzigen Standpunkte durch die Beschreibung seiner Sammlung versteinerter und fossiler Überreste des Thier- und Pflanzenreichs der Vorwelt erläutert. Gotha. 15 pls.
Vinn, O. 2006. Possible cnidarian affinities of Torellella (Hyolithelminthes, Upper Cambrian, Estonia). Paläontologische Zeitschrift 80 (4): 383-388.
Yochelson, E. L. 1964. Book review: The Tentaculites of Bohemia: Their morphology, taxonomy, phylogeny and biostratigraphy. Journal of Paleontology 39 (3): 509-510.