It all depends on what the ball is sitting on.In the absence of background information, a film of a tennis ball hitting a 1 ton block of concrete at 5mph would not enable you tell whether one, the other or both were moving. However, I presume that you would not expect the tennis ball to suffer the same amount of damage if it were dropped at 5mph onto the concrete block as it would if the concrete block were dropped at 5mph onto the ball - or would you?!
If the ball is sitting on a surface that can barely hold it's weight and a heavy concrete block is dropped on it at 5mph i'd expect the ball to be fine. The surface supporting the ball would break and the ball would be accelerated so it was moving at a speed greater than that of the concrete block.
On the other hand if the ball is sitting on a concrete driveway which is effecitvely an immovable object then things are going to end up much worse for the ball.
Similarlly in a low speed collision between a person and a car i'd expect the biggest hazard would not be from being accelerated by the impact but from being caught between the car and something that is effectively an "immovable object".