Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I use AES128 crypto in CTR mode for encryption, implemented for different clients (Android/Java and iOS/ObjC). The 16 byte IV used when encrypting a packet is formated like this:

<11 byte nonce> | <4 byte packet counter> | 0
The packet counter (included in a sent packet) is increased by one for every packet sent. The last byte is used as block counter, so that packets with fewer than 256 blocks always get a unique counter value. I was under the assumption that the CTR mode specified that the counter should be increased by 1 for each block, using the 8 last bytes as counter in a big endian way, or that this at least was a de facto standard. This also seems to be the case in the Sun crypto implementation.
I was a bit surprised when the corresponding iOS implementation (using CommonCryptor, iOS 5.1) failed to decode every block except the first when decoding a packet. It seems that CommonCryptor defines the counter in some other way. The CommonCryptor can be created in both big endian and little endian mode, but some vague comments in the CommonCryptor code indicates that this is not (or at least has not been) fully supported:
http://www.opensource.apple.com/source/CommonCrypto/CommonCrypto-60026/Source/API/CommonCryptor.c
/* corecrypto only implements CTR_BE.  No use of CTR_LE was found so we're marking
   this as unimplemented for now.  Also in Lion this was defined in reverse order.
   See <rdar://problem/10306112> */
By decoding block by block, each time setting the IV as specified above, it works nicely.
My question: is there a "right" way of implementing the CTR/IV mode when decoding multiple blocks in a single go, or can I expect it to be interoperability problems when using different crypto libs? Is CommonCrypto bugged in this regard, or is it just a question of implementing the CTR mode differently?
                How many bits get included in the counter seems to vary a lot. Some use 64 bits, others the whole 128 bits. But that shouldn't have any effect in your case, since the LSB is 0 and you never have more than 256 blocks per IV.
– CodesInChaos
                Sep 21, 2012 at 11:34
                @CodesInChaos indeed, and the chance that you encrypt more than 2^64 blocks is not that high either, so the top 64 bits will be unaffected.
– Maarten Bodewes
                Sep 21, 2012 at 14:25
                @owlstead 2^32 packets are enough to potentially cause a roll over to the highest bits in the OP's scheme. Still that's pretty unlikely. In schemes where the whole IV is random, even the second block might cause a difference between 128 bit and 64 bit counters.
– CodesInChaos
                Sep 21, 2012 at 14:28
                @CodesInChaos: I agree, but I hope for the sake of the asker that the package number is fully unique, and that it will never contain more than blocksize * 256 bytes. Seems to me that somebody is broken in the CommonCrypto implementations, or that the library is used differently than intended (finalizing the crypto too early instead of updating).
– Maarten Bodewes
                Sep 21, 2012 at 14:33
                @owlstead Yes, they are unique, and there should be no case of reusing the same IV, unless the random nonce is reused by chance. Also, the key is never reused, so every new nonce comes with a new random key.
– Markus R
                Sep 21, 2012 at 19:35
The definition of the counter is (loosely) specified in NIST recommendation sp800-38a Appendix B. Note that NIST only specifies how to use CTR mode with regards to security; it does not define one standard algorithm for the counter.
To answer your question directly, whatever you do you should expect the counter to be incremented by one each time. The counter should represent a 128 bit big endian integer according to the NIST specifications. It may be that only the least significant (rightmost) bits are incremented, but that will usually not make a difference unless you pass the 2^32 - 1 or 2^64 - 1 value.
For the sake of compatibility you could decide to use the first (leftmost) 12 bytes as random nonce, and leave the latter ones to zero, then let the implementation of the CTR do the increments. In that case you simply use a 96 bit / 12 byte random at the start, in that case there is no need for a packet counter.
You are however limited to 2^32 * 16 bytes of plaintext until the counter uses up all the available bits. It is implementation specific if the counter returns to zero or if the nonce itself is included in the counter, so you may want to limit yourself to messages of 68,719,476,736 = ~68 GB (yes that's base 10, Giga means 1,000,000,000).
because of the birthday problem you've got a 2^48 chance (48 = 96 / 2) of creating a collision for the nonce (required for each message, not each block), so you should limit the amount of messages;
if some attacker tricks you into decrypting 2^32 packets for the same nonce, you run out of counter.
In case this is still incompatible (test!) then use the initial 8 bytes as nonce. Unfortunately that does mean that you need to limit the number of messages because of the birthday problem.
                Note that reading the spec, you can be reasonably sure that NIST only uses Big Endian mode, so it's not so surprising that Apple only opted for that one in the end. Remarks like that in a crypto library do make me shudder though, it seems to be an indication that the implementer did not read the specifications, or did not understand them. In that case, they should not be allowed to create a crypto library.
– Maarten Bodewes
                Sep 21, 2012 at 14:23
                Thanks for the info. I will read up on it and see if I can get CommonCrypto to work like it is supposed to, or otherwise file a bug with Apple.
– Markus R
                Sep 21, 2012 at 19:39
                The use case here is an encrypted UDP stream, with the IV and key being sent once via an encrypted TCP side channel. I need the packet counter to construct the IV, since packets may be lost.
– Markus R
                Sep 21, 2012 at 19:42
                I don't agree with the statement that with an 8 byte random part "you've got a chance of 2^32 to have a collision in the nonces". To be more correct, on average it will take 2**32 messages before you see a repeated keystream. However, you will want to stop using the key way before that. For CCM (which uses CTR), RFC5084/CMS recommends 12 bytes for the nonce. Other sources are in the same ballpark. 8 bytes is not secure as a general recommendation.
– SquareRootOfTwentyThree
                May 6, 2013 at 5:21
                Changed my recommendation according to the remark of @SquareRootOfTwentyThree  Although this has a slightly higher chance of incompatibilities  compared to an 8 byte nonce, I think the increase of security is worth it in this case. If an 8 byte nonce is required by the library make sure you limit the number of messages encrypted with the key.
– Maarten Bodewes
                Apr 4, 2015 at 12:25
Further investigations sheds some light on the CommonCrypto problem:
In iOS 6.0.1 the little endian option is now unimplemented. Also, I have verified that CommonCrypto is bugged in that the CCCryptorReset method does not in fact change the IV as it should, instead using pre-existing IV. The behaviour in 6.0.1 is different from 5.x.
This is potentially a security risc, if you initialize CommonCrypto with a nulled IV, and reset it to the actual IV right before encrypting. This would lead to all your data being encrypted with the same (nulled) IV, and multiple streams (that perhaps should have different IV but use same key) would leak data via a simple XOR of packets with corresponding ctr.
                The current CC headers say for CCCryptorReset: Use only for CBC mode. And if you look at the source, you will see that it checks for macOS 10.13/iOS 11 or newer and if not, it falls back to a compatibility function that ignores the fact that calling ccSetIV in fact returns kCCParamError. Look for CCCryptorReset_binary_compatibility opensource.apple.com/source/CommonCrypto/…
– Mecki
                May 2, 2019 at 16:54
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.