Pennies weren't always synonymous with a paltry sum. Early in their existence, they were made from silver and often cut into pieces. Even after pennies transitioned to copper, they still weren't the smallest denomination. In fact, it wasn't until the 1970s that the ha'penny began to be phased out in England. Several countries stopped issuing pennies or their equivalent over the years. Canada officially stopped in 2012, and many people interested in the matter have suggested that the United States do so as well.
Crazy? Not so much. It currently costs the U.S. more to produce a penny than it's worth. We're losing money every time a new penny shows up in circulation. Still, that doesn't mean their worthless, as this video shows.