Sir Phineas Ryrie, the first chairman of the Hong Kong Jockey Club in 1884, was also chairman of the General Chamber of Commerce and a Legislative Councillor from 1867 until his death in 1892. He originally came from Stornoway in the remote, far north of Scotland. He was one of three brothers famed for their connection to the China trade. Sir Phineas established his business reputation as a trader in Hong Kong, initially with Turner & Co. Together with his partner Alexander Finlay Smith, who had previously worked for Scotland's Highland Railway, Sir Phineas founded the High Level Tramway Company to build the Peak Tram. Sir Phineas had built himself a mansion on The Peak, which he called "Craig Ryrie".

A socially conscious businessman, Sir Phineas was renowned for promoting freedom of speech in the Legislative Council. He also spoke out against legalising gambling, which makes his appointment as the inaugural Jockey Club chairman somewhat surprising, despite his known love of racing. He was a sporting enthusiast. He introduced rabbits to Stonecutters' Island so he could shoot them. But the rabbits failed to reproduce. Perhaps they knew Sir Phineas was watching. His two brothers made their names at sea. One brother, Alexander, drowned in 1855, when his ship, Jardine Matheson's Audax, was lost with all hands during a typhoon en route from Shanghai to Hong Kong. The older brother John Ryrie was famous as the captain of Jardine Matheson's big tea clipper, the "Cairngorm".