First of all, the Church doesn't understand the Genesis account literally. You are free to take it that way, if you wish, but if you spend time trying to interpret it literally, you'll miss a good deal of the theology taught.
That said, the narrative allows for a literal interpretation. Adam and Eve also had many other sons and daughters. It only names another son, Seth, but it does say Adam and Eve had many sons and daughters.
Genesis 4: Beginnings of Civilization
25 And Adam knew his wife again, and she bore a son and called his name Seth, for she said, "God has appointed for me another child instead of Abel, for Cain slew him." 26 To Seth also a son was born, and he called his name [Enosh|Enos]. At that time men began to call upon the name of the LORD.
Genesis 5: Adam's Descendants to Noah and His Sons
1 This is the book of the generations of Adam. When God created man, he made him in the likeness of God. 2 Male and female he created them, and he blessed them and named them Man when they were created. 3 When Adam had lived a hundred and thirty years, he became the father of a son in his own likeness, after his image, and named him Seth. 4 The days of Adam after he became the father of Seth were eight hundred years; and he had other sons and daughters.
(Genesis 4:25 to Genesis 5:4)
So, if you insist on taking the text literally, there's your answer.