Near-infrared (NIR) photodetectors sensitive to 860 nm were fabricated through a solution process with two different conjugated polymers (CPs) sharing a similar conjugated backbone consisting of the diketopyrrolopyrrole (DPP) moiety. The two CPs had comparable conjugated nature, but their detection capability was significantly distinguished by the applied linker units, thiophene for PDPP-T and fluorinated thiophene for PDPP-FBT. First, it has been revealed that employing additive during active film preparation is beneficial to achieve optimum film morphology for both CPs, which can suppress dark current and provide enhanced detectivity. In addition, when a highly electronegative fluorine atom that induces a surface dipole was introduced into the conjugated skeleton, it has been clearly demonstrated by transient photovoltage measurement that the charge recombination time is lengthened, which implies the increase of possibility of charge extraction in actual device through reducing the charge recombination. Thus, PDPP-FBT exhibited higher external quantum efficiency and responsivity overall wavelength. Our results indicate that the management of both film morphology and surface dipole of CP are evenly important to achieve high detectivity and responsivity in organic NIR photodetectors.