I haven't seen performance issues, but it's a relatively small deployment in the scheme of things. There are also existing solutions for caching here. NSCD seems to be the go-to for caching LDAP query results directly. Alternately, you could cache credentials at the PAM level with pam-ccreds (Debian package name).
