We probably would be able to. But consider that the "backdoor" can be incredibly sneaky. Look up the claimed "Crypto AG" attack, which involved pre-internet crypto devices. The claimed way it worked was to include the key or enough data to allow decryption in the transmitted message in some form.
NSA wouldn't need something to contact their data centres and leak the information, because presumably they can tap some fiber connection or other your data ends up travelling to. All they'd need would be a slow leak of your keys. Even, say, just a few of tweaked data per packet, or tacked on to a file format that is resistant to it, so that they can pick up key information together with the data stream, and they're good.
Or even just intentionally introducing a specific way of generating keys, known to the with sufficient precision to allow them to reduce the key space enough that brute forcing the remaining bits is feasible. For a conceptual example, look up the old Netscape SSL vulnerability (Netscape's SSL used to rely on factors like the host time and pid, that especially on a typical Unix-y system also running services like mail servers that often would include the pid of of the delivery agent in headers etc., was extremely easy to narrow down)
It's difficult to do this in a way that can't/won't sooner or later be discovered, but given the number of likely unintentional mistakes that have been made that still have taken in some cases years to be discovered (publicly at least), it's plausible that there are intentional flaws in at least some software to make it possible for specific parties to be intended to be able to break it.
NSA wouldn't need something to contact their data centres and leak the information, because presumably they can tap some fiber connection or other your data ends up travelling to. All they'd need would be a slow leak of your keys. Even, say, just a few of tweaked data per packet, or tacked on to a file format that is resistant to it, so that they can pick up key information together with the data stream, and they're good.
Or even just intentionally introducing a specific way of generating keys, known to the with sufficient precision to allow them to reduce the key space enough that brute forcing the remaining bits is feasible. For a conceptual example, look up the old Netscape SSL vulnerability (Netscape's SSL used to rely on factors like the host time and pid, that especially on a typical Unix-y system also running services like mail servers that often would include the pid of of the delivery agent in headers etc., was extremely easy to narrow down)
It's difficult to do this in a way that can't/won't sooner or later be discovered, but given the number of likely unintentional mistakes that have been made that still have taken in some cases years to be discovered (publicly at least), it's plausible that there are intentional flaws in at least some software to make it possible for specific parties to be intended to be able to break it.