In 1962 British Hong Kong, Shanghainese expatriates Chow Mo-wan, a journalist, and Su Li-zhen (Mrs. Chan), a secretary at a shipping company, rent rooms in adjacent apartments. Each has a spouse who works and often leaves them alone on overtime shifts. Due to the friendly but overbearing presence of Su's Shanghainese landlady, Mrs. Suen, and their bustling, mahjong-playing neighbours, Chow and Su are often alone in their rooms and rarely dine with the other tenants. Although they initially are friendly to each other only as need be, they grow closer as they realize that their spouses are having an affair with each other, and subsequently try to reenact how the affair might have begun.

Chow invites Su to help him write a martial arts serial. Their increased time together draws the attention of their neighbors, leading Chow to rent a hotel room where they can work together undistracted. As time passes, they acknowledge that they have developed feelings for each other. When Chow takes a job in Singapore, he asks Su to go with him. She agrees but arrives at the hotel too late to accompany him.

The next year, in Singapore, Chow relays a story to his friend about how in older times, when a person had a secret, they would go atop a mountain, make a hollow in a tree, and whisper it into the hollow and cover it with mud. Su arrives at Singapore and visits Chow's apartment. She calls Chow but remains silent when Chow picks up the phone. Later, Chow realizes she had visited his apartment after seeing a lipstick-stained cigarette butt in his ashtray.

Three years later, Su visits Mrs. Suen, who is about to emigrate to the United States, and inquires about whether her apartment is available for rent. Sometime later, Chow returns to Hong Kong to visit his former landlords the Koos, who have emigrated to the Philippines. He asks about the Suen family next door, and the new owner tells him a woman and her son are now living there. Chow leaves.

During the Vietnam War, Chow travels to Cambodia and visits Angkor Wat. As a monk watches him, Chow whispers something into a hollow in a wall and plugs it with mud.