On our Earthly pilgrimage, unnatural thoughts, especially of a sexual nature are part of the demon's war against us and our souls.
That said, the thoughts would not be sinful but acting on the thoughts would be sinful. If this is a hindrance in your spiritual life bring this issue up in Confession . . . regularly if needed.
If we strive to prevent them but give in, that's good; when we dwell on them purposely, that's going too far.