Support for Ikarus/LeGo #231
Comments
|
Hi, just to let you know: I'll be looking into this as well. I'm about to take a look at |
Haven't seen a sources anywhere, so fell free ;) |
|
If anyone finds this helpful too, here's a decompilation of The Chronicles of Myrtana: Archolos :) |
|
Just my 5 cents here: Is that really required to make OpenGothic compatible with all of that mem read mem write functions and anything that mess around with bytecode and/or internal class structures? Obviously due to many differences in native class structure mod code will not be able to produce same results as on original engine(and crash game in most cases). Maybe it worth to focus on compatibility layer that will allow to dynamically replace/hook/patch VM code and implement all required functionality directly in C++? For example, items picking while swimming in CoM: locate all places that messing with engine from VM and stub/hook them, implement feature as c++ plugin for opengothic, enjoy feature. So it will be possible to run some of the biggest and most noticeable mods with manual patching & using original mod distro. And maybe provide normal modding api for modmakers to ensure they are creating their mods on modern engine without messing with memory patching. |
Yeah, I was looking at that, but for now I concluded that it isn't that straightforward since we would have to replace mod-internal functions. We would have to replace all functions in CoM which do any kind of non-trivial direct memory access using Ikarus, which iirc were quite a lot, though I haven't been able to compile a full list. Additionally, any update to the mod then has the potential to break in OpenGothic, so it would likely require constant updates to the implementation. A more general implementation would be better, I think. I'm trying to figure out a good solution looking at CoM. Some things are trivial (like memory allocation) while others seem very hard (like mocking the ZenGin class structures). |
|
Hi, @lmichaelis , thank for bumping this up.
OK, but can you go into a few examples to illustrate problematic cases?
I was thinking about emulating memory-mapping-unit in engine, to mock ZenGin classes (at least to some extent) and implement only a few Ikarus api's. Basically implement layer that is close-most to DMA/Winapi and assume that rest will 'just work' |
|
Some examples: const int OCLOGMANAGER_PTR = 11191608;
func void LOG_MOVETOTOP(var string TOPICNAME) {
LIST = _^(OCLOGMANAGER_PTR);
NEWLISTPTR = LIST_CREATE(0);
NEWLIST = _^(NEWLISTPTR);
PREVLIST = _^(OCLOGMANAGER_PTR);
WHILE(LIST.NEXT);
LIST = _^(LIST.NEXT);
if (LIST.DATA) {
if (HLP_STRCMP(MEM_READSTRING(LIST.DATA), TOPICNAME)) {
NEWLIST.DATA = LIST.DATA;
PREVLIST.NEXT = LIST.NEXT;
};
};
PREVLIST = _^(_@(0x5446));
if ((NEWLIST.DATA) == (0)) {
if ((GAME_RUNTIMECONFIG) != (TARGET_SHIPPING)) {
MEMINT_FORCEERRORBOX = TRUE;
MEM_SENDTOSPY(ZERR_TYPE_FAULT, CS3("B_LogEntry: Error! Probably topic '", TOPICNAME, "' does not exist. Probably B_LogEntry has been called before Log_CreateTopic."));
MEMINT_FORCEERRORBOX = FALSE;
};
};
END;
LIST = _^(OCLOGMANAGER_PTR);
NEWLIST.NEXT = LIST.NEXT;
LIST.NEXT = NEWLISTPTR;
}func void TORCHLOADFIX_CHECKTORCH() {
if (!(HLP_ISVALIDNPC(HERO))) {
return;
};
PTR = TORCH_GETSLOTITEM();
if (PTR) {
ITM = _^(PTR);
if ((HLP_GETINSTANCEID(ITM)) == (0x87b7)) {
TORCHLOADFIX_GIVETORCH = TRUE;
};
};
if ((!(PTR)) || ((HLP_GETINSTANCEID(ITM)) != (0x87b7))) {
TORCHLOADFIX_GIVETORCH = FALSE;
};
}func int CATINV_NEXTNONACTIVEITEM(var int LIST, var int MAX) {
I = 0;
WHILE((LIST) && ((I) < (MAX)));
L = _^(LIST);
if (HLP_IS_OCITEM(L.DATA)) {
ITM = _^(L.DATA);
if (((ITM.FLAGS) & (ITEM_ACTIVE)) != (ITEM_ACTIVE)) {
BREAK;
};
I += 1;
};
LIST = L.NEXT;
END;
return I;
}instance ITGR_MEATFRIED(C_ITEM) {
NAME = FOODNAME_MEATFRIED;
DESCRIPTION = NAME;
HP = _@(ITEMGROUP_MEAT[0]);
VALUE = MAX_IG_MEATFRIED;
}
instance ITGR_VEGETABLE(C_ITEM) {
NAME = FOODNAME_VEGETABLES;
DESCRIPTION = NAME;
HP = _@(ITEMGROUP_VEGETABLE[0]);
VALUE = MAX_IG_VEGETABLE;
}There are a lot more direct uses of This is why we can't easily just replace Ikarus and LeGo functions since usage of these |
As mentioned there is a compatibility class ptr32_t at = mem.alloc(sizeof(phoenix::c_item)); // allocate fake region
mem.pin(at, this->hitem.get()); // pin it to real c++ memory
...
mem.free(at); // doneWith this setup, any invalid access should be trapped in |
|
Looks like that most of the functions can be replaced entirely(like log_movetotop). Others - some array access, which can be implemented relatively easy. IDK if maintenance is a problem, I don't think that CoM will be changed drastically, and others actively developed mods most likely will be glad to use normal open-sourced engine and implement their features without messing around with zengin(I may be wrong, assuming how many old(and school as well) persons in gothic modding community). Sorry if that is an offtopic. |
Yes, but we still quite far away from point, when opengothic is expected to be main platform to run the game |
|
Done more KoM testing today: This is a bad news, because we can't really emulate *.dll for them :/ |
I failed to check the size of compiled Gothic scripts of KoM (files seem 0 bytes in my installation of the game), so cannot compare to the size of .dll. Maybe you will be able to do so, or maybe contacting the devs of KoM would be a good idea. What I'm getting to is, perhaps the part of KoM functionality that has been placed in .dll is relatively small in compare to the .d scripts they created. In such a case, perhaps it would be relatively small effort to translate the .dll part to something that OpenGothic can handle. For example, if OpenGothic introduces API for modders (I think I saw this being planned in the future), like LUA scripts or whatever, then maybe KoM devs or anyone could translate the .dll to OpenGothic API. I think there could be more mods that are created partly in .d and prtly in external code that is injected to Gothic process with Ikarus. Once you introduce some modern API for moders to OpenGothic, perhaps it would be good idea to also introduce something to your dedalus VM that allows to inject scripts/code written in the API. So that .d and API can be integrated to some point. That might make it easier to switch Ikarus based mods to OpenGothic. Not sure if that makes perfect sense... Btw. I'm curious how KoM injects the .dll code to Gothic process. I understand Ikarus is able to make Gothic execute x86 code from any memory address, but how do they load DLL code to a memory space available to Gothic? Is it also done with Ikarus, like copying bytes from file to memory? |
Well, I can tell you exactly what is there: So, steam related stuff + music
Ikarus enables calls to C++ functions, so just call
This is far term stuff. introducing something like LUA as a scripting language will create segmentation in mods community, what is not good. |
|
Some up-to date progress: Can launch Chronicles of Myrtana, with only 7-screens of errors in log. Only early game is playable, dialog and cut-scene when we have to leave ship does not start. Shorten log: |
* CoM: initialization routine in progress #231 * CoM: loops and more support for 32-bit memory #231 * ikarus/lego in progress #231 * build * fix LeGo::create * phoenix::func #231 * build * update mem-traps * mem arrays * fix for LeGo::create * opaque_memory support #231 * memory_instance in progress * transient_instance update * rework start-points * Update phoenix
|
finally fix start-point determination: |
Subject
Ikarus is a popular library that provides set of DMA(direct memory access) utilities, exploring volubility in vanilla script-interpreter.
LeGo is a game framework bases on Ikarus, that provides all sort of goods, like custom-views, painting, AI, and more.
Ikarus strongly relies on how exactly original game works: addresses of
oGame/zTimer/oCInformationManagerare hard-coded, pointer are stored inint32- something what is not doable on x64 platformGoal
Create good-enough compatibility layer, to enable most popular mods:
the Chronicles of Myrtana,VarusBiker Editionand othersDesign
We won't provide same memory layout for most of things, so support won't be bullet-prof. Instead, OpenGothic will give access to high-level functional.
To make int-pointers work OpenGothic will provide a fake 3GB heap structure, with layout like virtual memory in win32.
Mappings:
Memory layout should resemble Win32 heap, while be sealizable:
Api(Ikarus):
Base on ikarus_Doc.d
JW - mean 'just works' - function does not require engine-level support.
Access will be reliable only for manually allocated memory(
MEM_Alloc), engine-memory generally won't do, unless there is no alternative.Memory
int MEM_ReadInt(var int address)void MEM_WriteInt(var int address, var int val)string MEM_ReadString(var int address)void MEM_WriteString(var int address, var string val)int MEM_ReadIntArray (var int arrayAddress, var int offset)[JW]int MEM_WriteIntArray (var int arrayAddress, var int offset, var int value)[JW]int MEM_ReadByte(var int adr)[JW]void MEM_WriteByte (var int adr, var int val)[JW]Parser
instance MEM_PtrToInst(var int ptr)void MEM_AssignInst(var int inst, var int ptr)instance MEM_CpyInst (var int inst)Functions & Reflection
void MEM_CallByString(var string fnc)void MEM_CallByID(var int ID)void MEM_PushIntParam(var int param)void MEM_PushStringParam(var string strParam)void MEM_PushInstParam(var int instance)int MEM_PopIntResult()string MEM_PopStringResult()instance MEM_PopInstResult()int MEM_GetFuncID(var func function)void MEM_Call(var func function)int MEM_FindParserSymbol (var string inst)int MEM_GetParserSymbol (var string inst)Note: Symbols do exist in OpenGothic, but do not match in terms of memory-layout
Jumps & Control flow
MEM_InitLabelsMEM_StackPos.positionStrings
int STR_GetCharAt(var string str, var int pos)int STR_Len (var string str)int STR_Compare (var string str1, var string str2)int STR_ToInt(var string str)string STR_SubStr(var string str, var int start, var int count)string STR_Prefix (var string str, var int count)[JW]string STR_Upper(var string str)int STR_SplitCount(var string str, var string Seperator)string STR_Split(var string str, var string Separator, var int index)int STR_IndexOf(var string str, var string tok)Menu
int MEM_GetMenuByString(var string menu)int MEM_GetMenuItemByString(var string menuItem)Global instances
instance MEM_Game (oCGame);instance MEM_World (oWorld);instance MEM_Timer (zCTimer);instance MEM_WorldTimer (oCWorldTimer);instance MEM_Vobtree (zCTree);instance MEM_InfoMan (oCInfoManager);instance MEM_InformationMan (oCInformationManager);instance MEM_Waynet (zCWaynet);instance MEM_Camera (zCCamera);instance MEM_SkyController (zCSkyController_Outdoor);instance MEM_SpawnManager (oCSpawnManager);func void MEM_InitGlobalInst()Ini file
string MEM_GetGothOpt (var string sectionname, var string optionname)string MEM_GetModOpt (var string sectionname, var string optionname)int MEM_GothOptSectionExists (var string sectionname)int MEM_GothOptExists (var string sectionname, var string optionname)int MEM_ModOptSectionExists (var string sectionname)int MEM_ModOptExists (var string sectionname, var string optionname)void MEM_SetGothOpt (var string section, var string option, var string value)Keyboard settings
int MEM_GetKey(var string name)int MEM_GetSecondaryKey(var string name)int MEM_KeyPressed(var int key)int MEM_KeyState(var int key)void MEM_InsertKeyEvent(var int key)Execute x86 code from memory
Won't do
Calls to engine functions
void CALL_IntParam(var int param)void CALL_FloatParam(var int param)void CALL_PtrParam(var int param)void CALL_zStringPtrParam(var string param)void CALL_cStringPtrParam (var string param)void CALL_StructParam(var int ptr, var int words)void CALL__stdcall(var int adr )void CALL__thiscall(var int this, var int adr)void CALL__cdecl(var int adr)void CALL__fastcall(var int ecx, var int edx, var int adr)int CALL_RetValAsInt()int CALL_RetValAsPtr()instance CALL_RetValAsStructPtr()string CALL_RetValAszStringPtr()void CALL_RetValIsFloat()int CALL_RetValAsFloat()void CALL_RetValIsStruct (var int words)void CALL_RetValIszString()string CALL_RetValAszString()Note: only fixed subset of function can be supported in individual manner
int LoadLibrary(var string lpFileName)int GetProcAddress (var int hModule, var string lpProcName)Note: generally won't do, due to portability complexity.
int MEM_SearchVobByName (var string str)int MEM_SearchAllVobsByName (var string str)int MEM_InsertVob(var string vis, var string wp)void MEM_TriggerVob (var int vobPtr)void MEM_UntriggerVob (var int vobPtr)void MEM_RenameVob (var int vobPtr, var string newName)int Hlp_Is_oCMob(var int ptr)int Hlp_Is_oCMobInter(var int ptr)int Hlp_Is_oCMobLockable(var int ptr)int Hlp_Is_oCMobContainer(var int ptr)int Hlp_Is_oCMobDoor(var int ptr)int Hlp_Is_oCNpc(var int ptr)int Hlp_Is_oCItem(var int ptr)int Hlp_Is_zCMover(var int ptr)int Hlp_Is_oCMobFire(var int ptr)void MEM_SetShowDebug (var int on)string MEM_GetCommandLine()int MEM_MessageBox (var string txt, var string caption, var int type)void MEM_InfoBox (var string txt)int MEM_GetSystemTime()int MEM_BenchmarkMS(var func f)int MEM_BenchmarkPC(var func f)int MEM_BenchmarkMS_N(var func f, var int n)int MEM_BenchmarkPC_N(var func f, var int n)int MEM_ReadStatArr (var int array, var int offset)void MEM_WriteStatArr (var int array, var int offset, var int value)int MEM_GetStringAddress(var string s)int MEM_GetFloatAddress (var float f)int MEM_GetIntAddress (var int i)int MEM_ArrayCreate ()void MEM_ArrayFree (var int zCArray_ptr)void MEM_ArrayClear (var int zCArray_ptr)void MEM_ArrayInsert (var int zCArray_ptr, var int value)void MEM_ArrayRemoveIndex (var int zCArray_ptr, var int index)void MEM_ArrayRemoveValue (var int zCArray_ptr, var int value)void MEM_ArrayRemoveValueOnce (var int zCArray_ptr, var int value)void MEM_CopyBytes (var int src, var int dst, var int byteCount)void MEM_CopyWords (var int src, var int dst, var int wordcount)[JW]void MEM_SwapBytes (var int ptr1, var int ptr2, var int byteCount)void MEM_SwapWords (var int ptr1, var int ptr2, var int wordcount)int MEM_CompareBytes (var int ptr1, var int ptr2, var int byteCount)int MEM_CompareWords (var int ptr1, var int ptr2, var int wordcount)void MEM_InitAll()int MEM_GetFuncPtr(var func fnc)int MEM_GetFuncOffset(var func fnc)int MEM_GetClassDef (var int objPtr)string MEM_GetClassName (var int objPtr)int MEM_GetBufferCRC32 (var int buf, var int buflen)int MEM_GetStringHash (var string str)int MEM_Alloc (var int amount)int MEM_Realloc (var int oldptr, var int oldsize, var int newsize)void MEM_Free (var int ptr)void MEM_SetParser(var int parserID)void MemoryProtectionOverride (var int address, var int size)- won't doThe text was updated successfully, but these errors were encountered: