ip: only set rx_sw_if_index when connection found to avoid following crash like tcp...