Although $x/0 is technically not infinity in a purely mathematical sense, when you understand why the IEEE float includes a value for infinity, and returns infinity in this case, it makes sense that PHP would agree with this.
The reason is that programmers don't usually divide by 0 on purpose. A value of 0 as a divisor usually occurs due to underflow, i.e. a value which is too small to be represented as a float. So, for example, if you have values like:
$x = 1;
$y = 1e-15 * 1e-15;
$z = $x/$y;
Because $y will have underflowed to 0, the division operation will throw the division by zero warning, and $z will be set to INF. In a better computer, however, $y would not have the value 0 (it would be 1e-30) and $z would not have the value INF (it would be 1e30).
In other words, 0 is not only representative of an actual 0, but also a very small number which float cannot represent correctly (underflow), and INF does not only represent infinity, but also a very big number which float cannot represent correctly (overflow). We do the best we can within the limitations of floating point values, which are really just good approximations of the actual numbers being represented.
What does bother me is that division by zero is handled in two different ways depending on the operator. I would have preferred the new DivisionByZeroError exception to be thrown in all cases, for consistency and to enforce good programming practices.