Only one person is definitely going to St Ives: the narrator.

What this refers to

Your phrase “how many are going to St Ives” is almost certainly referencing the classic riddle:

“As I was going to St Ives,
I met a man with seven wives,
Each wife had seven sacks,
Each sack had seven cats,
Each cat had seven kits…”

People often try to multiply all the wives, sacks, cats, and kittens to get a big number, but the trick is in the wording: only the speaker is explicitly said to be “going to St Ives.” Everyone else is just “met” on the way and may be traveling in the opposite direction or elsewhere. So the clean answer to “how many are going to St Ives?” is: one.