I've recently learned about certificates on Windows, and I must say collecting the information about them wasn't that easy. So I want to write down the summary while it's fresh in my mind. It's not the most exhaustive treatment but I hope that it's a good quick introduction.
First, what is a certificate? I hope that you already know the general idea, so here is only the short summary. A certificate is a public key along with the information about who it belongs to, cryptographically signed by someone you already trust.
A side note: in case if you don't know what the public key is, it's a part of asymmetric cryptography. Instead of encrypting and decrypting the data with the same key (as is done in the symmetric cryptography), the asymmetric cryptography uses the keys in the form of specially generated pairs. You keep one key of the pair private, and publish another one. The data encrypted with one key can only be decrypted with another key of the pair. It cannot be decrypted with the same key that was used for the encryption. So if you have someone's public key, you can encrypt some data with it, send it to the key owner, who can decrypt it with the private key. The concept of signing works the other way around: someone encrypts the data with the private key and publishes it. The public key gets also published but usually separately (the same key can be used to sign multiple messages). Then anyone can decrypt the data with the public key, and thus conclude that it really was encrypted by someone who knew the private key. There are great many details to all this, but this is the gist. If you want more details, get a popular book on cryptography. In particular, the concept of "chain of trust" is useful for understanding the certificates.
Returning back from the side note, the proper certificate as such contains only the public key, the information about its owner, and the signature on this information done by the certificate authority. It's everything that gets published. There is also a piece of information that doesn't get published, the private key. The public gets the certificate. The owner of the certificate has both the certificate and the private key. Kind of confusingly, on Windows this pair (certificate proper + private key) is also called a certificate. Looks like Microsoft has started doing it first but then the others have also picked up this tradition.
So there are "certificates without the private key" and "certificates with the private key". When you distribute the keys, you really need to make sure that you distribute only the certs without the private key.
The certificates need to be formatted as bytes. The standard X.509 defines a widespread kind of certificates, what's their meaning, formatting, and the algorithms and protocols involved. The company of the inventors of the asymmetric encryption, RSA, also has its own set of standards. They are called the PKCS standards. The standard PKCS#7 describes the way to format the certificate proper (the public information only). The standard PKCS#8 describes the way to format the private key. And the standard PKCS#12 defines a way to format both the public and the private part (though the private part is optional).
The standard PKCS#12 is also known as the "PFX file format". Microsoft had developed it first for its files with extension .pfx, and later it was adopted as the PKCS standard. That's why many of the Microsoft APIs refer to it interchangeably as "PFX" or "PKCS12".
In Windows, the certificates are kept in the Certificate Store. Fundamentally, it's kept in Registry but you're not supposed to mess with that part of Registry directly. There are APIs that do it for you. They're smart enough to keep the data in a complicated format that makes it difficult to steal your certs by someone who gets access to your physical hard drive. Even impossible if the OS is configured to use the Trusted computing with a special helper hardware module (TPM). You can also export the certificates from the Cert Store to a file, and import them back, but for the OS to use a cert, it has to be in the Cert Store.
To reiterate, it's important to not give away your private key. So when you export a certificate to a file to give it to someone else, use the format PKCS#7. It contains only the public information. The format PFX (AKA PKCS#12) is the one when you want to move your cert from one machine to another your machine. And to further safeguard the private keys, you can mark a cert in the cert store as "the private key is not exportable". Then you won't be able to export the cert in the PKCS#12 format at all.
Windows can use multiple encryption algorithms, so a certificate also includes the information about what algorithm it's for. When you use the crypto API, the appropriate algorithm is loaded from a DLL. This algorithm is formally known as a "Cryptography Provider".
Not all the crypto providers are equal. There are the "Crypto providers" as such and the "Key Store providers". The key store providers are a more recent addition, and the idea behind them is to make the private keys more secure. With a key store provider, Windows uses the public key internally but won't let the applications use it directly. You can still export the cert with a key store provider into a file, including the private key in it, and import it back but you can't get the direct access to the key.
If your goal is to create a cert for the machine authentication by WinRM with the HTTPS connection, a cert with a key store provider is fine for this purpose because it's something that the internals of Windows do for you. It's actually the better form of a cert for this purpose because the private key is slightly more secure.
But if your goal is to create a cert for direct encryption, a cert with a key store provider won't work. You'd be able to encrypt with the public key from it but you won't be able to decrypt with the private key from it because you won't get the access to the private key. For that you need a cert with the real crypto provider.