"The town's supposed to be where the center of the fighting is - you really want to walk into that all torn up?"

Guts' concern is to get them there in one piece and still ready to fight. This was only the start of whatever conflict they were being thrown into.