Forum

#1 2017-10-28 11:20:03

Narann
Member

Reverse engineering on CLuaCurve

Hello! For the guerilla_parser, we try to reverse engineer the data structure for curve.

Code:

LUICClassCreate("CLuaCurve","\001\002\002\000\000\000\000\224\004E\000\000@BUU\141\193\000\000PAUU\141A\000\000P\193\002\018\018\000\0000\008E\000\000\016AUU\141\193\000\000PAUU\141A\000\000P\193\002\018\018\000")

As you guess, it will be hard so it would be really appreciate to give some hint on how this hexadecimal content is structured.

Something like:

int (32bits) the number of key/value
float (32bits) key
float (32bits) value
short (8bits) tangeant type
float (32bits) key
float (32bits) value
short (8bits) tangeant type
float (32bits) key
float (32bits) value
short (8bits) tangeant type
etc.

That would really help us! smile

Thanks in advance!

Offline

 

#2 2017-10-28 13:50:49

Ben
Guerilla dev, the guy to hug

Re: Reverse engineering on CLuaCurve

Oh...
That is going to be fairly tough! The easiest is that we actually write these things in a human readable fashion.
Is this supposed to be in the file, or something you'd like to use in scripts?

Ben


Benjamin 'Ben' Legros
Guerilla developer
http://www.guerillarender.com

Offline

 

#3 2017-10-30 15:08:20

Narann
Member

Re: Reverse engineering on CLuaCurve

That is going to be fairly tough!

Yet another reason to justify my wage. {B-[

The easiest is that we actually write these things in a human readable fashion.

Of course but that would break compatibility.

Is this supposed to be in the file, or something you'd like to use in scripts?

Boths I guess. For now I'm looking for reading curves written in gproject files, modify them, and reencode them to put them back in guerilla project.
With this I could have a way to read/write values without opening Guerilla (which is the point of guerilla_parser). Of course, as guerilla_parser allow you to do anything, you can break your gproject (disconect things, inconsistent values) but you have to be smart. The point of the parser is mostly to modify frames/paths and stuff like this.

Offline

 

#4 2017-11-15 16:10:05

lpequignot
New member

Re: Reverse engineering on CLuaCurve

Hi Dorian,

I think I have a beginning of a solution smile

For example, with following curve data :

LUICClassCreate("CLuaCurve","\001\002\010\000\000\000\000\224\203D\000\000HB\000\000@\192UUU@\000\000@@UUU\192\002\018\018\000\000\000\205D\000\000 B\000\000@\192\243\026J@UUU@\184\143`\192\002\018\018\000\000@\206D\000\000\240AUUU\192UU5@UUU@UU5\192\002\018\018\000\000\128\207D\000\000\184AUUU\192UU\245?UUU@UU\245\191\002\018\018\000\000\192\208D\000\000\148AUUU\192\170\170\202?UUU@\170\170\202\191\002\018\018\000\000\000\210D\000\000XAUUU\192\000\000\192?UUU@\000\000\192\191\002\018\018\000\000@\211D\000\000\024AUUU\192\000\000\128>UUU@\000\000\128\190\002\018\018\000\000\128\212D\000\000@AUUU\192UU\245\191UUU@UU\245?\002\018\018\000\000\192\213D\000\000\168AUUU\192UUU\192UUU@UUU@\002\018\018\000\000\000\215D\000\000\000BUUU\192\170\170j\192UUU@\170\170j@\002\018\018\000")


001\002 # magic number ??
\010 = 10 # nb keys
\000\000\000 # format ?

\000\224\203D =1631.0 \000\000HB  =50.0 # key, value
\000\000@\192 =-3 UUU@ =3.3333332538605 \000\000@@  =3 UUU\192 =-3.3333332538605 # tangent in x,y / tangent out x,y
\002\018\018\000 # flags (unified, smooth + lock, smooth + lock, - )

\000\000\205D =1640.0 \000\000 B  =40.0
\000\000@\192 =-3 \243\026J@ =3.1578948497772 UUU@ =3.3333332538605 \184\143`\192 =-3.5087718963623
\002\018\018\000

\000@\206D = 1650.0 \000\000\240AUUU = 30.0
\192 UU5@ = 2.8333332538605 UUU@ =3.3333332538605 UU5\192 =-2.8333332538605
\002\018\018\000

[...]

I used following function to get float from str

function convertfloat(str)
  -- Change to b4,b3,b2,b1 to unpack an LSB float
  local b4,b3,b2,b1  = string.byte(str, 1, 4)
  print(b4, b3, b2, b1)
  local exponent = (b1 % 128) * 2 + math.floor(b2 / 128)
  if exponent == 0 then return 0 end
  local sign = (b1 > 127) and -1 or 1
  local mantissa = ((b2 % 128) * 256 + b3) * 256 + b4
  mantissa = (math.ldexp(mantissa, -23) + 1) * sign
  return math.ldexp(mantissa, exponent - 127)
end

ex :
convertfloat('\000\224\203D') -> 1631.0

and for the flags, maybe

0
1 selected=1
2 unified=2
3 selected=1 + unified=2

16 smooth=16 + unlock=0
18 smooth + lock=2
32 flat=32 + unlock=0
34 flat=32 + lock=2

48 linear=48 + unlock= 0
50 linear=48 + lock =2

The problem is, for now, I can't get the same result in lua and python
# ex : '\000\224\203D'
local (b4,b3,b2,b1)  = string.byte(str, 1, 4) will return ( 0, 224, 203, 68) in lua
and in python :
struct.unpack('>bbbb', '\000\224\203D')
(0, -108, -125, 68)


louise

Last edited by lpequignot (2017-11-16 06:45:21)

Offline

 

#5 2017-11-16 13:22:35

Ben
Guerilla dev, the guy to hug

Re: Reverse engineering on CLuaCurve

Damn ...
Thsi is some reverse engineering smile

Here is the current format of the curves, in binary:

Code:

# start stream

uint8    Version = 1
uint8    CurveFlags

uint32    Nkeys
uint32    NkeysCRC = hash (NKeys)

repeat Nkeys times:
    float    X, Y        key position
    float    PX, PY        previous tangent position
    float    NX, NY        next tangent position
    uint32    KeyFlags

# end stream


with:

CurveFlags = bitset
    Loop = 0x01
    ConstraintX = 0x02

KeyFlags = bitset
    KeySelected = 0x1
    TangentUnified = 0x2

    TangentPSelected = 0x100
    TangentPNormalize = 0x200    // Manual + normalize

    TangentPMode =   0xf000        // Mask mode
    TangentPManual = 0x0000        // Fully manual
    TangentPSmooth = 0x1000        // Smooth
    TangentPFlat =   0x2000        // Flat
    TangentPLinear = 0x3000        // Linear

    TangentNSelected =  0x10000
    TangentNNormalize = 0x20000

    TangentNMode =   0xf00000
    TangentNManual = 0x000000
    TangentNSmooth = 0x100000
    TangentNFlat =     0x200000
    TangentNLinear = 0x300000
    TangentNStep =     0x400000


uint32    hash (uint32 a)
{
   a = (a+0x7ed55d16) + (a<<12);
   a = (a^0xc761c23c) ^ (a>>19);
   a = (a+0x165667b1) + (a<<5);
   a = (a+0xd3a2646c) ^ (a<<9);
   a = (a+0xfd7046c5) + (a<<3);
   a = (a^0xb55a4f09) ^ (a>>16);
   return a;
}

Note that the string in the file is ascii encoded binary such that \xxx represents the xxx byte (e.g. \023 is the byte 23)
Multibyte values (float, uint32) are little endian.

So your stream looks like:

\001: version = 1
\002: CurveFlags = ConstraintX
\010\000\000\000: Nkeys = 10
\000\224\203D\000: NkeysCRC

    \000HB\000    X
    \000@\192U   Y
    UU@\000       PX
    \000@@U      PY
    UU\192\002   NX
    \018\018\000\000 NY
    \000\205D\000 KeyFlags

    ...

I hope I didn't mistranslate from the streaming code smile

Ben


Benjamin 'Ben' Legros
Guerilla developer
http://www.guerillarender.com

Offline

 

#6 2017-11-16 16:15:44

lpequignot
New member

Re: Reverse engineering on CLuaCurve

Hello Ben,

Thanks for the full solution !
That helps a lot smile

Offline

 

#7 2017-11-16 18:26:04

Ben
Guerilla dev, the guy to hug

Re: Reverse engineering on CLuaCurve

Otherwise, I'd be happy to give you Lua/Python functions to pack/unpack curves ...
Just let me know, I'll do that one evening smile

Ben


Benjamin 'Ben' Legros
Guerilla developer
http://www.guerillarender.com

Offline

 

#8 2017-11-18 12:23:09

Narann
Member

Re: Reverse engineering on CLuaCurve

Awesome job Louise. wink

And thanks Ben to give us the whole structure! This will really helps. big_smile

EDIT: Oh! If you want to provide us the Python function to unpack, of course we would love it. I will put it into guerilla parser! big_smile

Last edited by Narann (2017-11-18 12:30:52)

Offline

 

#9 2017-11-20 15:46:06

Narann
Member

Re: Reverse engineering on CLuaCurve

I tried to unpack curve values.

Here is a curve with a value of 0 at time 1 and value 5 at time 5. I try to parse the first key values:

Code:

>>> curve = "\001\002\002\000\000\000\000\000\128?\000\000\000\000\171\170\170\191VU\213\191\171\170\170?VU\213?\002\018\018\000\000\000\160@\000\000\160@\171\170\170\191VU\213\191\171\170\170?VU\213?\002\018\018\000"
>>> version, flags, n_keys, n_keys_hash, x, y, px, py, nx, ny, key_flags = unpack_from('<BBIIffffffI', curve)
>>> version
1
>>> flags
2
>>> n_keys
2
>>> n_keys_hash
940179456
>>> x
8.828180325246348e-44
>>> y
2.015849675747936e+34
>>> px
48714597007360.0
>>> py
0.00012354303908068687
>>> nx
2.0158557416541286e+34
>>> ny
-4.1087192164877637e-32
>>> key_flags
939590207

I guess X and Y should be both 0.0 no? Am I missing something? From your code:

Code:

float    X, Y        key position

And later:

Code:

    \000HB\000    X
    \000@\192U   Y

X is 2 bytes (16bits)? Y is 2 bytes too? But a float is 4 bytes (32bits).

Does this mean we have two float in one float? (half?)

Offline

 

#10 2017-11-20 16:02:31

Narann
Member

Re: Reverse engineering on CLuaCurve

Even playing with numpy I can't get my value 5:

Code:

>>> curve = "\001\002\002\000\000\000\000\000\128?\000\000\000\000\171\170\170\191VU\213\191\171\170\170?VU\213?\002\018\018\000\000\000\160@\000\000\160@\171\170\170\191VU\213\191\171\170\170?VU\213?\002\018\018\000"
>>> for i in range(200):
    print numpy.frombuffer(buffer(curve[i:i+2]), dtype=numpy.float16)[0]

    
3.0577e-05
3.0637e-05
1.1921e-07
0.0
0.0
0.0
0.0
0.00018311
0.50488
1.8047
3.7551e-06
0.0
0.0
0.0
40960.0
36640.0
36608.0
2.2411e-05
0.62549
0.16321
99.063
85.375
-0.00022376
2.3544e-05
0.62549
0.16321
42528.0
36640.0
36608.0
1.8672
99.938
85.375
-0.00022376
1.8857
3.4273e-05
1.5378e-05
0.50049
1.8597e-05
0.50049
3.3379e-06
0.0
0.0
8192.0
2.2188
3.8147e-06
0.0
8192.0
2.2188
43008.0
36640.0
36608.0
2.2411e-05
0.62549
0.16321
99.063
85.375
-0.00022376
2.3544e-05
0.62549
0.16321
42528.0
36640.0
36608.0
1.8672
99.938
85.375
-0.00022376
1.8857
3.4273e-05
1.5378e-05
0.50049
1.8597e-05
0.50049
3.3379e-06

Last edited by Narann (2017-11-20 16:03:13)

Offline

 

#11 2017-11-20 18:07:41

lpequignot
New member

Re: Reverse engineering on CLuaCurve

Hi,

As soon as you set the str, escaped number are converted, for example :
\12 -> \n
\171 -> x
\170 -> x
>>> "\001\002\002\000\000\000\000\000\128?\000\000\000\000\171\170\170\191VU\213\191\171\170\170?VU\213?\002\018\018\000\000\000\160@\000\000\160@\171\170\170\191VU\213\191\171\170\170?VU\213?\002\018\018\000"
'\x01\x02\x02\x00\x00\x00\x00\x00\n8?\x00\x00\x00\x00yxx\x0191VU\x8b\x0191yxx?VU\x8b?\x02\x018\x018\x00\x00\x00p@\x00\x00p@yxx\x0191VU\x8b\x0191yxx?VU\x8b?\x02\x018\x018\x00'
>>>

It's why I can't have the same in python /lua previously
# ex : '\000\224\203D'
local (b4,b3,b2,b1)  = string.byte(str, 1, 4) will return ( 0, 224, 203, 68) in lua
and in python :
struct.unpack('<bbbb', '\000\224\203D')
(0, -108, -125, 68)
struct.unpack('<f', '\000\224\203D')
(1052.625,)

but as Ben said  (\xxx represents the xxx byte)
\000 > 0
\224 > 224
\203 > 203
ord('D')
68
-> (0, 224, 203, 68)

>>> def convertfloat(b4,b3,b2,b1):
...     exponent = (b1 % 128) * 2 + math.floor(b2 / 128)
...     if exponent == 0 :
...        return 0
...     sign = (b1 > 127) and -1 or 1
...     mantissa = ((b2 % 128) * 256 + b3) * 256 + b4
...     mantissa = (math.ldexp(mantissa, -23) + 1) * sign
...     return math.ldexp(mantissa, long(exponent - 127))
...

>>> convertfloat(0, 224, 203, 68)
1631.0

Last edited by lpequignot (2017-11-20 18:08:04)

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2008 PunBB