Yes, you can brute force this rather quickly, but in case you are curious, you don't need to brute force this entire solution. There is a mathematical approach.
You might even require pieces of this anyway since this seems to be a hashing algorithm and not an encryption algorithm (I.e. it is not reliably reversible).
Does it make you curious that a repeatable encryption algorithm uses random number generation? That can only work when the random seed (look up
srand) is set consistently for the same input, meaning consecutive calls to
rand() will return the same ordered sequence of results.
Q. How is the random seed set in this case?
A. By the product of the character codes of the first four characters, the ones that happen to be missing.
We know from the code that the resulting encrypted text will have the same length as the plaintext input. We also know that each encrypted output character is generated by the original character at its index like this:
(random number generated from the seed at the n'th pass where n is the current index + 1) ^ (that same random number mod the key length of 13 in your example) ^ (the character code at the current index).
From that you can derive a simple mathematical formula to find
r at a given pass through encryption based on an output character code and index.
I couldn't find an official source for free, but several places like here show how the c
rand function is implemented:
static unsigned long int next = 1;
int rand(void) // RAND_MAX assumed to be 32767
next = next * 1103515245 + 12345;
return (unsigned int)(next/65536) % 32768;
void srand(unsigned int seed)
next = seed;
Notice that the seed is adjusted each time a number is generated. You can build another formula based on the
rand implementation to find what possible values
next can when
rand returns the character code associated with the 5th character of the output (the first one associated with a key character that you know). Take each of those possibilities and filter them down by running subsequent passes through
rand based on the code above and the encryption code using the
rand output as the
r value and checking the results against the next characters of the output until you are left with only a single valid possible option for
next at the 5th pass.
Once you know that, reverse that one time for each of the previous characters to get the first 4 values of
r. Once you have solved for those values of
r, all that's left is to reverse
output[i] = r ^ key[(r % strlen(key))] ^ text[i] to solve for
text[i]. It will be useful to recall that the inverse of
y = a ^ x is
y = log base a (x). Plug in the
r values you solved for previously in the right order to get the character codes associated with that pass through the encryption. Add those to the beginning of the key and check your work by reversing the encryption algorithm and passing in your new key with the encrypted text and seeing if the output makes sense. If it does, you've completed the challenge without brute force (except perhaps in your filtering).
Of course, since the key space is small enough to bruteforce, you can always do that.