That's the first thing they said. But I decided to look more into it.
I read something that some races are usually called ponies while others are usually not. But they also said that a race that's usually referred as a horse and not a pony can sometimes be called a pony if it's very small. Though, at the same time, there are races that are short enough to be called ponies but are still called horses.
Then again. There are more differences than just the hight. Apparently the muscle structure in a pony makes them way stronger, at least for their size. Ponies are often more tolerant to cold weather as well. Some studies apparently say that a pony is more intelligent than an average horse. They usually have better endurance, too.
There are more examples, but I don't feel like mention every single one. At the end of the day, saying a pony is not a horse is like comparing a Chiwawa to a Rottweiler and say that only one of them is a dog. Both are dogs, just like a pony is a horse. A special group of horses, sure, but still a horse.