Author Topic: Usage of GFS_NwFread (you probably expected this...)  (Read 739 times)

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Usage of GFS_NwFread (you probably expected this...)
« on: May 30, 2018, 12:27:14 am »
Hey,

I've got the GFS_NwFread function and structure from the Workshop 95 slides, and it compiles and "runs".
Of course what I am trying to do is use it to read XL2's ZTP files, which have segmented data meant for a modelData_t struct.

It seems XL2 loads these in segments with GFS_Load, loading one segment has information relevant to loading the next segment.
Of course I'm here because I don't understand how exactly that works, I'm guessing the reason this doesn't work with GFS_NwFread is because I'm not arranging the memory addresses right.
I say that because it doesn't load any of the other models using the current Address variable called after my dysfunctional '"ztFileStream" function is run.

I imagine all I should need to do is read from the buffer into the address for a modelData_t struct...
..
but i don't know how to do that...

/e: Oh, you should probably know that the function is in ZT_LOAD_MODEL.c  in ZT folder.

Code: [Select]
//Experimental
void * ztFileStream(Sint8 * filename, void * startAddress, entity_t * model)
{

    memset_l((void*)startAddress,0x0000, (0xFFFE - ((Uint32)startAddress-(Uint32)LWRAM)));  //Not 100% necessary, since data can just be overwritten, but good for testing and see how much data a level takes
    void * currentAddress;
    currentAddress = startAddress;

    Sint32 fid = GFS_NameToId((Sint8*)filename);
GfsHn gfs;
Sint32 stat;
Sint32 nsct;
Sint32 bsize;
//Model Data is the destination of data.
modelData_t bufModel;
//Destination address?
    void * ptr = &bufModel;

//Loading follows
gfs = GFS_Open(fid);
//Get sectors
GFS_GetFileSize(gfs, nsct, NULL, NULL);
bsize = 2048 * nsct;
// nsct = GFS_ByteToSct(gfs, bytesOf);

//modelData_t has an assumed size?
GFS_NwFread(gfs, nsct, (Uint32*)currentAddress, bsize);

//For loop is blank because this will run until GFS_Close, AKA always run until GFS_Close.
for(;;)
{
stat = GFS_NwExecOne(gfs);
if(stat == GFS_SVR_COMPLETED)
{
break;
}
//Copy gfs to modelData_t bufmodel???
memcpy_l((Sint32*)ptr, (Sint32*)(startAddress), (sizeof(modelData_t)));

/**ADDED**/
model->pos[X]=bufModel.Origin[X]; model->pos[Y]=bufModel.Origin[Y]; model->pos[Z]=bufModel.Origin[Z];
model->length[X]=bufModel.Length[X]; model->length[Y]=bufModel.Length[Y]; model->length[Z]=bufModel.Length[Z];
model->nbMeshes=bufModel.TOTAL_MESH;

//Get textures from GFS
Uint16 first_texture = loadTextures(startAddress, &bufModel);
//Get PDATA from GFS
// Sint32 bytesOff = (bufModel.TEXT_SIZE+(sizeof(modelData_t)))/2048;

// bytesOff = bufModel.TEXT_SIZE+(sizeof(modelData_t)) - (bytesOff*2048);
// currentAddress = (void*)(currentAddress + bytesOff);

currentAddress = loadPDATA((void*)currentAddress, model, &bufModel);
//Set textures from GFS
setTextures(first_texture, model, bufModel.TOTAL_MESH, true);

    return currentAddress;
}
GFS_Close(gfs);

}
« Last Edit: May 30, 2018, 12:29:56 am by ponut64 »

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #1 on: May 30, 2018, 01:05:24 am »
I didn't take a huge look.
For the function, I never used it myself, but I think the Jo Engine async loading uses it, so you should look there.
You don't need to load small segments since I aligned all the model data to just be dumped in memory, which is way faster.
You also return before closing the file, which is wrong.

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #2 on: May 30, 2018, 01:36:07 am »
Returning before closing the file.. yeah that one is on me, I should have known that would halt the function and thus halt the CD system on that file.
That change, plus commenting everything else out, allows everything else to load as normal.

Jo engine ASYNC loading does in fact use NwFread so I will look that over.

/e
It looks to me whether using what I've written or what Jo engine does, I'd just need to get the read data where its supposed to go.
Could be handled in the callback in the jo engine async read function or in that part of the written code between break and GFS close.
« Last Edit: May 30, 2018, 01:53:10 am by ponut64 »

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #3 on: May 30, 2018, 06:04:19 am »
Upon further reading, I can't think of why this would not work.
What happens? It does not load. All loading stops.

Code: [Select]
//Experimental
void * ztFileStream(Sint8 * filename, void * startAddress, entity_t * model, Sint8 dummy)
{

    memset_l((void*)startAddress,0x0000, (0xFFFE - ((Uint32)startAddress-(Uint32)LWRAM)));  //Not 100% necessary, since data can just be overwritten, but good for testing and see how much data a level takes
    void * currentAddress;
    currentAddress = startAddress;

    Sint32 fid = GFS_NameToId((Sint8*)filename);
GfsHn gfs;
Sint32 stat;
Sint32 nsct;
Sint32 bsize;
//Model Data is the destination of data.
modelData_t bufModel;
//Destination address?
    void * ptr = &bufModel;

//Loading follows
gfs = GFS_Open(fid);
//Get sectors
GFS_GetFileSize(gfs, &nsct, NULL, NULL);
bsize = 2048 * nsct;
// nsct = GFS_ByteToSct(gfs, bytesOf);

//modelData_t has an assumed size?
GFS_NwFread(gfs, nsct, (Uint32*)currentAddress, bsize);

//For loop is blank because this will run until GFS_Close, AKA always run until GFS_Close.
for(;;)
{
stat = GFS_NwExecOne(gfs);
if(stat == GFS_SVR_COMPLETED)
{
break;
}
//Copy gfs to modelData_t bufmodel???
memcpy_l((Sint32*)ptr, (Sint32*)(startAddress), (sizeof(modelData_t)));

    /**ADDED**/
    model->pos[X]=bufModel.Origin[X]; model->pos[Y]=bufModel.Origin[Y]; model->pos[Z]=bufModel.Origin[Z];
    model->length[X]=bufModel.Length[X]; model->length[Y]=bufModel.Length[Y]; model->length[Z]=bufModel.Length[Z];
    model->nbMeshes=bufModel.TOTAL_MESH;

    /**Load the texture list (using an offset to allow DMA transfer)**/
    Uint16 first_texture = loadTextures(startAddress, &bufModel);

    /**Load PDATA**/
    Sint32 bytesOff = bufModel.TEXT_SIZE+(sizeof(modelData_t)) - (bytesOff*2048);
    currentAddress = (void*)(currentAddress + bytesOff);

    currentAddress = loadPDATA((void*)currentAddress, model, &bufModel);

    /**Set textures**/
    setTextures(first_texture, model, bufModel.TOTAL_MESH, true);


}
GFS_Close(gfs);
    return currentAddress;

}

So what do we do:

1 - we set a new address so we don't overwrite the last data
2 - that address is now the current address
3 - we get a file ID from the file name
4 - variable and struct setup
? I question why bufModel is modelData_t and not entity_t, tried it one way or another it doesn't function properly
5 - Set GFS and GFS handler
6 - Get the file size in sectors, we don't need anything else (they are NULL).
6 - Set the size in bytes based on the sector size
7 - Queue the command NwFread for the file with current Address as the buffer area
8 - Execute NwFread in the open gfs
9 - When the server is complete with the file, stop the execution and
10 - memcpy destination bufModel, the source startAddress (which has never been set separate from currentAddress at this point BUT setting it to current causes unknown code crash), the size copied is the size of... you know.
? Why would current crash when it is no different from start...
? It doesn't crash or halt if everything past memcpy before GFS_Close is commented out.
11 What follows,
taking the contents of the buffered model data and putting it into the model which is the desired entity_t struct.

Truly, I know I don't really understand what is happening. Especially not enough to make a callback that could put the binary data into an entity_t struct from the Jo engine async read function.
I'm thinking in a forum post.

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #4 on: May 30, 2018, 06:31:00 am »
I didn't look at it for very long, but you put everything in your loop.
It loads textures and everything you don't need and loops these functions over, causing crashes.
In think you should just ditch the loop, load data until textures like I did, then start the async loading using the sector offset that I already had.
Then just load all the data in the cd buffer, wait until it's done and then fetch the data to your start address.
You could then call the other functions.
I never played with streaming functions, but the cd ram buffer can hold 512 KB (crazy that they put so much ram there and the Saturn still had crappy video support).
For your animations, it just wouldn't work, but you could try with images, maybe allowing you some kind of pseudo Resident Evil game where you could always cache the next backgrounds in RAM and transfert it to VRAM when you swap backgrounds.

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #5 on: May 30, 2018, 09:51:00 pm »
You know the way I was doing isn't technically how I wanted to do it anyway. I want to use the CD Buffer, and for that I need to first use NwCdRead then fetch from CD buffer into the work area.
I guess that's one way to potentially use the CD buffer as some sort of RAM.

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #6 on: May 31, 2018, 04:21:20 am »
I'm getting close to have animation support.
The new version and code should be out in a few days if all goes well (I could encounter deal breaking bugs as I know very little about animation).
DAE would have made my life way easier if I just focused on that format a while ago, but now I won't have time for it, so let's hope everything works OK!

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #7 on: May 31, 2018, 04:48:28 am »
My man, I'm sure I speak for a lot of folks who may not even post here. I really appreciate what you do, and what Jo has done for all of us wanting to make games for the Saturn.
Binary model files of any kind was a huge step toward creating a fully fledged 3D game for the Saturn.. rather modularly
I hadn't any time to work on what I was doing today either, but at least I can write code that compiles

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #8 on: May 31, 2018, 07:51:53 pm »
Thinking out here again. I'm debugging things a little more closely. I've come to this.

It reads the correct amount of data to the CD buffer.
It completes and does not crash.
memcpy successfully copies the CD buffer data to the struct buffer from ptr.
The data copied is correct.
After setting the entity_t data from the modelData_t buffer using your commands, the data is correct.
But after it finishes loading binaries, it is gone. The mesh does not have any data, whether or not I load something else.

I am missing something very simple... something that if I understood it, I could probably use jo engine async reading system.

Quote
//Experimental
void * ztFileStream(Sint8 * filename, void * startAddress, entity_t * model, Sint8 dummy)
{
   
    memset_l((void*)startAddress,0x0000, (0xFFFE - ((Uint32)startAddress-(Uint32)LWRAM)));  //Not 100% necessary, since data can just be overwritten, but good for testing and see how much data a level takes
    void * currentAddress;
    currentAddress = startAddress;
//Data of read structure
    Sint32 fid = GFS_NameToId((Sint8*)filename);
   GfsHn gfs;
   Sint32 nsct;
   Sint32 fsize;
//Data of read-in-process
   Sint32 stat;
   Sint32 rdsize;
//Model Data is the destination of data in the work area.
   modelData_t bufModel;
//Destination address?
    void * ptr = &bufModel;
//Loading follows
   gfs = GFS_Open(fid);
//Get sectors
   GFS_GetFileSize(gfs, &nsct, NULL, NULL);
   GFS_GetFileInfo(gfs, NULL, NULL, &fsize, NULL);
   
jo_printf(0, 18, "(%i)", nsct);
jo_printf(0, 20, "(%i)", fsize);
jo_printf(0, 22, "(%i)", proc_bp);

   
//Read from CD to CD buffer

   GFS_NwCdRead(gfs, fsize);
   GFS_SetTransPara(gfs, nsct);
   GFS_NwFread(gfs, nsct, (Uint32*)currentAddress, fsize);
   do{
      GFS_NwExecOne(gfs);
      GFS_NwGetStat(gfs, &stat, &rdsize);
jo_printf(0, 24, "(%i)", rdsize);
   }while (stat != GFS_SVR_COMPLETED && rdsize < fsize);
   
   GFS_Close(gfs);
   
   /**Copy gfs to modelData_t bufmodel???**/
   memcpy_l((Sint32*)ptr, (Sint32*)(currentAddress), sizeof(modelData_t));
   
    /**ADDED**/
    model->pos[X]=bufModel.Origin[X]; model->pos[Y]=bufModel.Origin[Y]; model->pos[Z]=bufModel.Origin[Z];
    model->length[X]=bufModel.Length[X]; model->length[Y]=bufModel.Length[Y]; model->length[Z]=bufModel.Length[Z];
    model->nbMeshes=bufModel.TOTAL_MESH;
   
    Uint16 first_texture = loadTextures(startAddress, &bufModel);
    Sint32 bytesOff = bufModel.TEXT_SIZE+(sizeof(modelData_t)) - (bytesOff*2048);
   currentAddress = (void*)(currentAddress + bytesOff);
   currentAddress = loadPDATA((void*)(currentAddress), model, &bufModel);
   currentAddress = startAddress;
    setTextures(first_texture, model, bufModel.TOTAL_MESH, true);
   
   jo_printf(0, 14, "(%i)", bufModel.TOTAL_MESH);
   jo_printf(0, 16, "(%i)", bufModel.PDATA_SIZE);
   jo_printf(0, 10, "(%i)", model->nbMeshes);
   
          return currentAddress;
   
}

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #9 on: May 31, 2018, 09:14:34 pm »
My functions overwrite the texture data after transfering to VRAM, so it doesn't work because you don't update the currentaddress correctly.
Look at your byte offset : you initialize it by calling itself, which doesn't work.
That offset is for cd sectors, which doesn't work here since all the data is dumped in RAM.
Your current adress should be : sizeof (mod data) + Texture size.
But it's bad practice because you don't need the textures in work RAM.

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #10 on: June 01, 2018, 07:48:53 pm »
I got it working now. For some reason though I need to put it in a function that is in a jo callback area (as in, run it when jo_core is running). /e: Do note that is the INTENDED USE, ANYWAY.
I still need to work in some refinements, like getting it to load sector-by-sector and file-by-file so it doesn't pause everything else.
I might even see about doing memcpy in smaller increments.

Some manner of smooth file streaming like that must be possible, not for any specific purpose, just a technical curiosity. Thanks for your help XL2. You were right about the offset.

Code: [Select]
//Experimental
void * ztFileStream(Sint8 * filename, void * startAddress, entity_t * model, Sint8 dummy)
{

    memset_l((void*)startAddress,0x0000, (0xFFFE - ((Uint32)startAddress-(Uint32)LWRAM)));  //Not 100% necessary, since data can just be overwritten, but good for testing and see how much data a level takes
    void * currentAddress;
    currentAddress = startAddress;
//Data of read structure
    Sint32 fid = GFS_NameToId((Sint8*)filename);
GfsHn gfs;
Sint32 nsct;
Sint32 fsize;
//Sector step of reading.
Uint8 rd_step = 2;
//Data of read-in-process
Sint32 stat;
Sint32 rdsize;

Sint32 i;

//Model Data is the destination of data in the work area.
modelData_t bufModel;
//Destination address?
    void * ptr = &bufModel;

//Loading follows
gfs = GFS_Open(fid);
//Get sectors
GFS_GetFileSize(gfs, &nsct, NULL, NULL);
GFS_GetFileInfo(gfs, NULL, NULL, &fsize, NULL);
//Buffers are the destination of data in the CD area.
Uint32 buf1[rd_step * nsct / 4];
Uint8 *proc_bp;

// jo_printf(0, 18, "(%i)", nsct);
// jo_printf(0, 20, "(%i)", fsize);
// jo_printf(0, 22, "(%i)", proc_bp);
// jo_printf(0, 24, "(%i)", rdsize);

//Read from CD to CD buffer

GFS_NwCdRead(gfs, fsize);
GFS_SetTransPara(gfs, nsct);
GFS_NwFread(gfs, nsct, (Uint32*)currentAddress, fsize);
do{
GFS_NwExecOne(gfs);
GFS_NwGetStat(gfs, &stat, &rdsize);

}while (stat != GFS_SVR_COMPLETED && rdsize < fsize);

GFS_Close(gfs);

/**Copy gfs to modelData_t bufmodel???**/
memcpy_l((Sint32*)ptr, (Sint32*)(currentAddress), sizeof(modelData_t));

    /**ADDED**/
    model->pos[X]=bufModel.Origin[X]; model->pos[Y]=bufModel.Origin[Y]; model->pos[Z]=bufModel.Origin[Z];
    model->length[X]=bufModel.Length[X]; model->length[Y]=bufModel.Length[Y]; model->length[Z]=bufModel.Length[Z];
    model->nbMeshes=bufModel.TOTAL_MESH;

Uint16 first_texture = loadTextures(currentAddress, &bufModel);
Sint32 bytesOff = bufModel.TEXT_SIZE+(sizeof(modelData_t));
currentAddress = (void*)(currentAddress + bytesOff);
currentAddress = loadPDATA((void*)(currentAddress), model, &bufModel);
currentAddress = (void*)(currentAddress + (bufModel.PDATA_SIZE + sizeof(modelData_t)));
setTextures(first_texture, model, bufModel.TOTAL_MESH, true);

// jo_printf(0, 6, "(%i)", bufModel->TOTAL_MESH);
// jo_printf(0, 8, "(%i)", bufModel->PDATA_SIZE);
// jo_printf(0, 12, "(%i)", model->nbMeshes);

    return currentAddress;

}

Of course, make sure this only runs once.
« Last Edit: June 01, 2018, 08:07:44 pm by ponut64 »

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #11 on: June 02, 2018, 02:24:50 pm »
On my end I got vertices animation export/import working, but the more I look at it the more I hate this.
While vertices animation is more flexible, it takes a lot of memory and it's quite complicated to set up using .OBJ files.
Again, I won't have time for Collada support, but it would be way better.
The main issue is that it's a complicated format and it takes forever to get it right.

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #12 on: June 03, 2018, 05:45:43 am »
Ok, I will support 2 ways to do animation : per vertex and by matrix.
The per vertex will require the user to set up his own table to know which frames to play.
The per matrix animation allows the user to manually set the meshes ID, so this way you can make all your models match the same IDs and keep one hierarchy format.
Then you will have to write your own animations, but it saves a lot of memory and you can reuse the same animations on several models.
Both ways require some work, but anyway I think it's nice to have 2 options.



ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #13 on: June 03, 2018, 07:36:26 am »
Very sensible. And exciting.

While I haven't gotten any further, I may have found out that the fetches (NwFread) are what bog the system down (A bus) and the system probably isn't waiting for these functions to complete to continue, they just bog down the A bus and they end up waiting for one another to complete since the A bus is busy. So it should be simple to schedule the fetch from CD buffer in a user-defined 'acceptable fetch chunk' per frame. More can be done.
« Last Edit: June 03, 2018, 07:38:26 am by ponut64 »

ponut64

  • Full Member
  • ***
  • Posts: 175
  • Karma: +13/-0
    • View Profile
Re: Usage of GFS_NwFread (you probably expected this...)
« Reply #14 on: June 07, 2018, 11:44:07 pm »
So I am working on problems for two potential solutions.

One, a more tightly controlled loading loop that can be timed to iterate no more than once per frame. The main issue with this is NwFread does not seem to function properly if NwExecOne or NwExecServer is in anything but a do... while loop, which is troublesome since that will always iterate once when the program passes over it. There are still more issues regarding just how massively slow the program runs when this loop is present, even if it isn't running. One thing I have identified is moving the control variables for the loop to global instead of local variables helps control the slowdown.

Two, a variant of how Jo Engine loads things adapted to handle ZTP files. This seems somewhat better but it still chugs the system when the file is first issued out, maybe there is no getting around that BUT it chugs the system MORE than if the single-file-load-loop is run to completion. That's why I really wish I could time out that single file load loop to iterate and run more smoothly, but alas!
Another problem with the jo engine load loop is it straight-up crashes on large files (like TITLE.ZTP included in some demos around here).

Attached is a file that has these two versions of code. ztServeRequests in the second variant is supposed to be a jo engine callback.

At this point I am more curious about how SGL loading functions. Reportedly slower, but I'd like to try them, unfortunately the commands are not in the compiler to do so.
« Last Edit: June 08, 2018, 12:05:56 am by ponut64 »

 

Sitemap 1 2 3 4 5 6 7 8 9 10 
SMF spam blocked by CleanTalk