Few studies have investigated the different effects that the built environment may have on the physical activity behaviours of men and women. Therefore, the aim of this study was to estimate the gender differences in meeting walking recommendations in relation to perceived neighbourhood walkability attributes within the active transportation and leisure-time domains for Czech inhabitants over 50 years of age. The sample included 1,417 men and 1,422 women who were randomly selected. The Abbreviated Neighbourhood Environment Walkability Scale (ANEWS) was used to obtain information about the perceived environment. The self-administered long version of the IPAQ was used to assess physical activity levels. When walking for transport, men living in neighbourhoods with high street connectivity (OR = 1.47, CI = 1.04–2.9) and higher traffic and crime safety (OR = 1.28, CI = 1.02–1.6) and women living in neighbourhoods with high proximity (OR = 1.36, CI = 1.04–1.77) and high neighbourhood aesthetics (OR = 1.36, CI = 1.04–1.76) were more likely to meet recommended levels of walking. No environmental attributes were found to significantly influence the accomplishment of walking recommendations by men or women when walking for leisure. The study results indicate the gender-specific associations between transportation-related walking and the environment factors. The consideration of those factors in the design of gender-specific walking interventions for Czech inhabitants may help the interventions to be more effective in promotion of physical activity.