The most common reason, given by a quarter of the males surveyed, was being too tired after work, while 19 percent of women said sex was too much of a hassle, he told Reuters.
"It's a question of work-life balance," Kitamura said in a telephone interview. "This is not something that the individual can tackle alone. The people who run companies need to do something about it."
His written survey involved 647 men and 821 women aged up to 49, who are married.
The trend could have serious consequences for Japan, whose falling birthrate and aging population are continuing headaches for the government.
The average number of children born to a Japanese woman in her lifetime was 1.34 in 2007, compared with 2.1 in the United States in 2006.
Kitamura said doctors may also be partly to blame for Japan's negative attitudes toward sex during pregnancy or after giving birth.
He is set to report to the Ministry of Health on his findings next year.
Is this possible? Can fatigued really lessen your sex-drive?