BEIJING—Richard Liu, the founder of e-commerce company JD.com, long labored in the shadows of his bigger and more famous rival Alibaba.

In early 2011, when JD was much smaller than it is now, Alibaba tried to persuade Mr. Liu to open a JD store inside Alibaba’s online marketplace instead of competing head on against the giant. If Mr. Liu took the offer, Alibaba would promote his business and help bring a lot of traffic. If he didn’t, JD would have to stop using Alibaba’s popular Alipay payment system, according to Mr. Liu. The talks between Alibaba and JD were previously unreported.