As explained in an earlier post (which you may be interested in reading as a bit of background to this one), the earlier part of the Caenozoic (the current era of the earth's history) was home to a number of mammalian lineages of very mysterious relationships. Very few of the familiar orders around us today had yet put in an appearance, and instead the world was home to such oddities as pantodonts, tillodonts and dinocerates. Among the prominent carnivorous mammals of the time were a group known as the creodonts. Creodonts ranged in size from that of a small cat to lion- or bear-size species, and often converged in appearance with those animals. But what were creodonts?
Current authors regard the Creodonta as including two families, the vaguely cat-like Oxyaenidae and the largely dog- or hyaena-like Hyaenodontidae. Oxyaenids were found in North America and Europe during the late Palaeocene and Eocene, while hyaenodontids were found in Africa, Eurasia and North America from the Late Palaeocene to near the end of the Miocene, though they disappeared from North America not long after the end of the Eocene (Gheerbrant et al., 2006). Many authors have suggested a relationship with modern carnivorans (cats, dogs, weasels, bears, etc.), and they have been included with the latter in a superorder Ferae. Popular as this arrangement has been, however, there's just one small problem - there's not a shred of evidence to support it.
Part of the problem is that creodonts are a good example of what might be called "taxonomic drift". Imagine that an author establishes a taxon, and presents a list of organisms that he thinks belong to that taxon. A few years pass by, and the taxon is revised by another author, who excludes some of the originally-included species that he thinks belong elsewhere, and substitutes a few more species that he believes to be related to the remainder. Carry this on through a few subsequent revisions, with species being taken out and put in, and you may end up with a situation where nearly all of the original members of the taxon have been taken out, and the taxon name has become associated with a very different concept from its original intent. This can be horrendously confusing for later readers, because if they don't realise that this taxonomic drift has taken place, they may read things into older publications that their authors never intended.
Creodonta was originally established by Edward Drinker Cope in 1875 as a suborder of the Insectivora*. In his new suborder, Cope included three families - Oxyaenidae, Ambloctonidae (now included in Oxyaenidae - Gunnell, 1998) and Arctocyonidae (another contemporary family of carnivorous placentals, within which Cope also included what are now regarded as the Miacidae). The Hyaenodontidae were not part of the original Creodonta - at the time, Hyaenodon was regarded as a genuine carnivoran. Cope distinguished creodonts from carnivorans by the former's lack of a fused scapholunar bone in the wrist, their ungrooved astragalus, and their less-developed and smoother cerebral hemispheres (Cope, 1884). These features, it should be noted, are all primitive for placentals, but to Cope indicated the creodonts' position in the insectivoran grade. He nevertheless regarded creodonts as ancestral to carnivorans, with cats descended from Oxyaenidae and dogs from Miacidae (Cope, 1880). Later, Cope (1883) included Insectivora and Creodonta as separate suborders of his order Bunotheria, which also included the tillodonts, taeniodonts and prosimians**. Cope (1883) also redefined creodonts to include mammals without continuously-growing incisors and with trituberculate molars, which meant that in addition to the Oxyaenidae and Miacidae, Creodonta now included Mesonychidae, Leptictidae, moles and tenrecs (Arctocyonidae were transferred to the Insectivora). The Hyaenodontidae wormed their way in a year later (Cope, 1884).
*This does not necessarily mean that he thought they were specifically related to modern insectivorans such as shrews and hedgehogs. Cope and most of his contemporaries would have regarded the "Insectivora" as representing the generalised basal form from which all other placental mammals were derived, and recent insectivorans would have been the remnants of that original grade.
**It is also notable that Cope regarded the aye-aye as forming a separate suborder from other prosimians, due to its rodent-like incisors. Cope (1884) held that the tillodonts were "intimately allied to the living Chiromys [aye-aye] of Madagascar, which is itself almost a lemur, by general consent" (emphasis mine).
So right from the beginning, the question of what was a creodont was convoluted. Over the years, various families of "creodonts" were reassigned as their relationships became clearer. The Miacidae became regarded as true Carnivora. Arctocyonidae and Mesonychidae became included among the primitive ungulates (another confused mess, but that's a story for another year) and may be related to artiodactyls. Moles and tenrecs, of course, were reunited with their fellow modern insectivorans (though the tenrecs have recently had another falling-out). Eventually, the creodonts were whittled down to their modern content of oxyaenids and hyaenodontids, but, as pointed out by Polly (1996), "Hyaenodontidae and Oxyaenidae are currently grouped together in Creodonta because they are the only taxa that have not been removed from the group, not because there has been specific positive evidence proposed for their grouping". Those few characters the two families do share are also found in other, non-creodont mammals. As for their association with Carnivora, the two orders have been associated because they both possess shearing carnassial teeth. However, while the carnassials in Carnivora are formed by the last upper premolars and the first lower molars, those of Oxyaenidae are derived from the first upper and second lower molars, while hyaenodontids have two sets of carnassials formed by the first upper/second lower and second upper/third lower molars. Carnassials have also developed in other groups of mammals - notably the borhyaenoids, which are metatherians if not marsupials and so definitely not related to carnivorans. The only real reason creodonts have been associated with Carnivora for so long seems to be their prior inclusion of the genuinely carnivoran (or stem-carnivoran) miacids. It's a bit like when one of your friends brings an acquaintance of theirs to a party who just hangs around for hours with everybody being too polite to ask them to leave.
So, if they weren't related to Carnivora, can we say what creodonts were related to? Particularly in the case of Oxyaenidae, the answer is brief, simple and to the point: we really have not got a sodding clue. Whatever their ancestry might have been, oxyaenids were horribly derived little (or not so little) beggars - for instance, they had completely lost the third molars. Van Valen (1969) derived both oxyaenids and hyaenodontids from the Palaeoryctidae, particularly from the Cretaceous-Palaeocene Cimolestes, and other authors seem to have regarded the idea favourably, at least for the hyaenodontids (Polly, 1996; Gheerbrant et al., 2006). The main problem with this scenario, however, is that the Palaeoryctidae of Van Valen and other authors is itself polyphyletic. For instance, the phylogenetic analysis of Wible et al. (2007) included two "palaeoryctids", Cimolestes and Eoryctes (Eoryctes is more likely to represent the Palaeoryctidae proper),and while Cimolestes appeared outside the placental crown group, Eoryctes was placed among the insectivorans as the sister to Potamogale (Tenrecidae). If creodonts (either or both families) are closer to Cimolestes, they may be stem-eutherians. If they are closer to Palaeoryctidae proper, they may even be afrotheres (Wible et al. did not support placement of tenrecs among afrotheres, but it is notable that the earliest hyaenodontids are African). Placement of either the Oxyaenidae or the Hyaenodontidae still awaits proper analysis.
Cope, E. D. 1880. On the genera of the Creodonta. Proceedings of the American Philosophical Society 19 (107): 76-82.
Cope, E. D. 1883. On the mutual relations of the bunotherian Mammalia. Proceedings of the Academy of Natural Sciences of Philadelphia 35: 77-83.
Cope, E. D. 1884. The Creodonta. American Naturalist 18 (3): 255-267.
Gheerbrant, E., M. Iarochene, M. Amaghzaz & B. Bouya. 2006. Early African hyaenodontid mammals and their bearing on the origin of the Creodonta. Geological Magazine 143 (4): 475-489.
Gunnell, G. F. 1998. Creodonta. In Evolution of Tertiary Mammals of North America vol. 1. Terrestrial Carnivores, Ungulates, and Ungulatelike Mammals (C. M. Janis, K. M. Scott & L. L. Jacobs, eds) pp. 91-109. Cambridge University Press.
Polly, P. D. 1996. The skeleton of Gazinocyon vulpeculus gen. et comb. nov. and the cladistic relationships of Hyaenodontidae (Eutheria, Mammalia). Journal of Vertebrate Paleontology 16 (2): 303-319.
Van Valen, L. 1969. The multiple origins of the placental carnivores. Evolution 23 (1): 118-130.
Wible, J. R., G. W. Rougier, M. J. Novacek & R. J. Asher. 2007. Cretaceous eutherians and Laurasian origin for placental mammals near the K/T boundary. Nature 447: 1003-1006.