[tedjpoole]

This commit removes the use of the RTLD_DEEPBIND option when the compat layer loads the OpenSSL shared libraries. As a replacement, the compat layer is now built with the -fvisibility=hidden option instead. This has the same effect of preventing OpenSSL code linking against the compat layer symbols instead of it's own. This change has been made, in preparation for adding an ASAN build of the compat layer, because the dlopen() wrapper that ASAN injects, does not work with the RTLD_DEEPBIND option.

Also added a test to confirm that compat layer symbols are not dynamically visible (preventing the subtle misresolution bugs that RTLD_DEEPBIND was originally introduced to solve).

[jwendell]

I don't think this test can fail as written. bssl-compat is linked statically into the test executable, and the test isn't linked with -rdynamic/--export-dynamic. So SSL_new/SSL_read/SSL_write never make it into the executable's dynamic symbol table, and dlsym(RTLD_DEFAULT, ...) returns nullptr even without -fvisibility=hidden (e.g. on main).

To make it a real regression test, could you add linkopts = ["-rdynamic"] to the utests-bssl-compat target in compat/openssl/test/BUILD? Then dropping -fvisibility=hidden from compat/openssl/BUILD should make this test fail. Worth checking that it does.

[jwendell]

Without RTLD_DEEPBIND, libcrypto's default CRYPTO_malloc/free should already resolve malloc/free through global scope to the executable's tcmalloc. So the "bypass tcmalloc" reason no longer really holds, and this call is now a safeguard. Maybe reword:

Suggested change

	// Tell OpenSSL to use the tcmalloc malloc/realloc/free functions from the
	// main executable. Without this, RTLD_DEEPBIND causes OpenSSL's allocations
	// to resolve to glibc's malloc/realloc/free instead, resulting in OpenSSL
	// using a completely separate heap. This defeats tcmalloc's performance
	// benefits on the TLS hot path, and also makes all OpenSSL allocations
	// main executable. OpenSSL uses its own allocator wrappers internally, so
	// without this, OpenSSL allocations bypass tcmalloc, defeating its
	// performance benefits on the TLS hot path and making OpenSSL allocations
	// invisible to tcmalloc's heap dumps.
	// Tell OpenSSL to use the tcmalloc malloc/realloc/free functions from the
	// main executable. Without RTLD_DEEPBIND, OpenSSL's default allocators
	// should already resolve to the executable's malloc via global scope, but
	// setting them explicitly guarantees OpenSSL uses tcmalloc regardless of
	// how symbol resolution behaves (e.g. under sanitizer interceptors).