from UCMP website
All chordates have the following features at some point in their life (in the case of humans and many other vertebrates, these features may only be present in the embryo):
pharyngeal slits - a series of openings that connect the inside of the throat to the outside of the "neck". These are often, but not always, used as gills.
dorsal nerve cord - a bundle of nerve fibers which runs down the "back". It connects the brain with the lateral muscles and other organs.
notochord - cartilaginous rod running underneath, and supporting, the nerve cord.
post-anal tail - an extension of the body past the anal opening.
"In bilateral invertebrates the nerve cord is ventral to the alimentary canal or gut... [vertebrates, in which the spinal cord is dorsal, the gut lying ventral to it]" [my brackets]
O'Shea, The Brain p.47