dragging map doesn't move click targets

[zodern]

I have a phaser game with a map made up of sprites that is 4,500 px by 4,500 px. Towns (every 5th sprite) can be clicked. The sprites are 250px square. 

 

I use: 

sprite.inputEnabled = true; 

sprite.events.onInputUp.add(function() { /* ... */}).bind({town: townId}, this);

 

When the map is dragged, the click targets stay in the same place. If I drag the map up and click where the town moved to won't do anything. Clicking where a town was at will call the click event for that town. How do I make the click targets move with the sprite?

[zodern]

If it helps, I am moving the camera when dragging.