Agreed. Inspiration can come from anywhere, at any time. Sometimes (rarely) a whole song seems to write itself from the smallest spark.
I keep a sloppy, haphazard file of ideas, bits of lyric, notes, chords etc that I occasionally revisit in case they have miraculously transformed into finished...