Author Topic: Reading binary files with pointers instead of returned char stream?  (Read 2970 times)

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
I managed to build my custom binary file format and read it on the Saturn, so I can display a mesh loaded from a binary file on the CD.

My concern now is that I'm using the jo_fs_read_file, which returns a char stream, which takes memory while I'm loading everything to the right place.
It's not a huge problem while loading small images and sound files, but for a map taking 1 MB, it just won't be doable (unless I'm mistaken and got it all wrong).

Is there a way, in Jo Engine, to just use a pointer to read the file and return each time only what you need (like 8, 16, 24 or 32 bits from a start pointer)?
If not, can it be added in the next version?
Thanks!

mindslight

  • Administrator
  • Full Member
  • *****
  • Posts: 157
  • Karma: +6/-1
    • View Profile
    • Jo Engine
Re: Reading binary files with pointers instead of returned char stream?
« Reply #1 on: August 08, 2017, 01:37:22 pm »
Sure :)

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #2 on: August 08, 2017, 10:52:11 pm »
Great! Thanks!

I know the topic is now in "Wish list", but I could need some help.
Bad news ; while the binary mesh loading works in Yabause, in SSF it's a real mess (but works with lower compatibility) while on real hardware it doesn't work at all (it crashes before even loading anything).
It's probably a mistake related to memory allocation or initalization, but I have no idea what's wrong at the moment.
I've attached the files, if anyone with better knowledge of memory allocation could look it up, I would really appreciate!
Thanks!

Code: [Select]
//Start address is 0x00200000

void * LoadBinaryMAP(void * startAddress)
{

 char * stream;
 void * currentAddress;
 int length;
 mesh_tot = 0;
 stream = jo_fs_read_file("MAPT.ZTM", &length);

register unsigned int idx = 0;
register unsigned short i=0;
register unsigned int ii=0;
register unsigned int iii=0;
register unsigned int nbpoint=0;
register unsigned int nbpolygon=0;

currentAddress = startAddress;

mesh_tot = jo_swap_endian_ushort(*((unsigned short *)(stream)));
idx+=2;

for (i=0; i<mesh_tot; i++)
{
    jo3dMesh[i] = currentAddress;
    nbpoint = jo_swap_endian_uint(*((unsigned int *)(stream+idx)));
    idx+=4;
    nbpolygon = jo_swap_endian_uint(*((unsigned int *)(stream + idx)));
    idx+=4;

    jo3dMesh[i]->data.pntbl = (POINT*)(jo3dMesh[i] + sizeof(unsigned int));
    for (JO_ZERO(ii); ii<nbpoint; ii++)
    {
        for (JO_ZERO(iii); iii<3; iii++)
        {
            jo3dMesh[i]->data.pntbl[ii][iii] =  jo_swap_endian_int(*((int *)(stream + idx)));
            idx+=4;
        }
    }
    jo3dMesh[i]->data.pltbl = (POLYGON*) (jo3dMesh[i]->data.pntbl + sizeof(POINT)*nbpoint);
    for (JO_ZERO(ii); ii<nbpolygon; ii++)
    {
        for (JO_ZERO(iii); iii<3; iii++)
        {
            jo3dMesh[i]->data.pltbl[ii].norm[iii] = jo_swap_endian_int(*((int *)(stream + idx)));
            idx+=4;
        }
        for (JO_ZERO(iii); iii<4; iii++)
        {
            jo3dMesh[i]->data.pltbl[ii].Vertices[iii]=jo_swap_endian_ushort(*((unsigned short*)(stream + idx)));
            idx+=2;
        }
    }
    jo3dMesh[i]->data.attbl = (ATTR*) (jo3dMesh[i]->data.pltbl + sizeof(POLYGON)*nbpolygon);

    for (JO_ZERO(ii); ii<nbpolygon; ii++)
    {
        jo3dMesh[i]->data.attbl[ii].texno = jo_swap_endian_ushort(*((unsigned short*)(stream + idx)));
        idx+=2;
        jo3dMesh[i]->data.attbl[ii].sort = SORT_MAX;
        jo3dMesh[i]->data.attbl[ii].flag = Dual_Plane;
        jo3dMesh[i]->data.attbl[ii].colno = No_Palet;
        jo3dMesh[i]->data.attbl[ii].gstb = No_Gouraud;
        jo3dMesh[i]->data.attbl[ii].dir = MESHoff, sprNoflip, UseLight;
    }

    jo3dMesh[i]->data.nbPoint = nbpoint;
    jo3dMesh[i]->data.nbPolygon = nbpolygon;

    currentAddress = (void*) (jo3dMesh[i] + sizeof(jo3dMesh[i]));
}
jo_free(stream);
jo_printf(0, 6, "current adress :            %d     ", currentAddress);
return currentAddress;
}
« Last Edit: August 10, 2017, 05:06:39 am by XL2 »

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #3 on: August 09, 2017, 03:42:15 pm »
The good folks at SegaXtreme helped with it, and it was an issue with data alignment.
Seems to be fixed for now!
I will update the tool and FPS demo in the next few days to show map loading from the disc.

mindslight

  • Administrator
  • Full Member
  • *****
  • Posts: 157
  • Karma: +6/-1
    • View Profile
    • Jo Engine
Re: Reading binary files with pointers instead of returned char stream?
« Reply #4 on: August 09, 2017, 04:04:12 pm »
Great !

BTW, I'll add more filesystem feature in the next release

mindslight

  • Administrator
  • Full Member
  • *****
  • Posts: 157
  • Karma: +6/-1
    • View Profile
    • Jo Engine
Re: Reading binary files with pointers instead of returned char stream?
« Reply #5 on: August 12, 2017, 03:16:16 pm »
Hi XL2,

You will find below, the prototypes of the functions I have added in the next release:

Code: [Select]
/** @brief Open a file
 *  @param file Pointer to an allocated jo_file struct
 *  @param filename Filename (upper case and shorter as possible like "A.TXT")
 *  @return true if succeed
 */
bool            jo_fs_open(jo_file * const file, const char *const filename);

/** @brief Close a file
 *  @param file Pointer to an allocated and valid jo_file struct
 */
void            jo_fs_close(jo_file * const file);

/** @brief Read bytes from a file
 *  @param file Pointer to an allocated and valid jo_file struct
 *  @param buffer Pointer to an allocated buffer (length >= nbytes)
 *  @param nbytes number of bytes to read
 *  @return Number of bytes read (<= 0 means EOF)
 */
int             jo_fs_read_next_bytes(jo_file * const file, char *buffer, unsigned int nbytes);


Sample:

Code: [Select]
    jo_file     file;
    char        buf[2048];

    if (jo_fs_open(&file, "TEST.TXT"))
    {
        while (jo_fs_read_next_bytes(&file, buf, 2048) > 0)
            /*DO SOMETHING */;
        jo_fs_close(&file);
    }

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #6 on: October 09, 2017, 04:17:54 am »
Would it be possible to add a function to read from an user-defined pointer?

Like, if I'm loading map 2 and it's using the same background as map 1, I want to be able to skip it and just keep it without loading it all over again.

So something like :
jo_fs_read_from_pointer(&file, buffer, StartingByte, Length);

Also, I've had a few failed read on real Saturn.
The Saturns are getting old, so I think it would be nice to implement a re-read feature in case of failure to read.
Like, in your read functions, if it fails, you try again.

Thanks!

mindslight

  • Administrator
  • Full Member
  • *****
  • Posts: 157
  • Karma: +6/-1
    • View Profile
    • Jo Engine
Re: Reading binary files with pointers instead of returned char stream?
« Reply #7 on: October 09, 2017, 08:37:31 am »
Noted  ;)

mindslight

  • Administrator
  • Full Member
  • *****
  • Posts: 157
  • Karma: +6/-1
    • View Profile
    • Jo Engine
Re: Reading binary files with pointers instead of returned char stream?
« Reply #8 on: October 11, 2017, 02:16:37 pm »
I made a new version (http://jo-engine.org/download/) with CD read-retry and a new function:

Code: [Select]
/** @brief Seek forward from current position of a file
 *  @param file Pointer to an allocated and valid jo_file struct
 *  @param nbytes number of bytes to skip
 *  @return true if succeed
 */
bool            jo_fs_seek_forward(jo_file * const file, unsigned int nbytes);

Now you can skip a part of a file and use jo_fs_read_next_bytes() after (just don't forget to shift the pointer : buffer + nbytes)

Example:

Code: [Select]
jo_file file;
char *file_contents;
...

jo_fs_open(&file, "FILE.TXT");
jo_fs_seek_forward(&file, 4096); /* I assume that 4096 is the size of the part to skip in the file and to preserve in the buffer */
jo_fs_read_next_bytes(&file, file_contents + 4096, 42);
jo_fs_close(&file);

« Last Edit: October 11, 2017, 02:22:01 pm by mindslight »

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #9 on: October 19, 2017, 02:51:42 am »
Thanks for the function, I'm using it already as I write palette data but I'm not using it (I tried to make 3d quads work with it by using the img_8_bits loading function, but it didn't go too well, so I'll have to write my own sprite loading function for 4 bits sprites textures).

Now, one more request (last one about this I hope) : instead of using a char for the buffer parameter, can you allow us to just put the memory address for direct loading?
I swapped the endianness on the PC side to support Saturn's endianness and I am using the jo_fs_read_next_bytes function to write directly to memory, which should speed up things nicely.
The only problem is that the compiler doesn't like this and throws warnings, which are annoying obviously, but otherwise it works fine.
So I'm not using a buffer at all, just straight to memory, and I think it would be nice that the function works this way right away (since I'm probably the only one using it at the moment, I don't think it will cause huge issues if you make the change on the existing function).
Thanks!

Danny

  • Newbie
  • *
  • Posts: 32
  • Karma: +3/-0
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #10 on: October 19, 2017, 02:35:07 pm »
Hi friend!
C is a strongly typed language so it enforces correct use of types, but if you know what you are doing you can cast a pointer to something else.
You can cast the return pointer to the type you are using like int* or your_type* or a void* (which matches everything) and the warning will go away.
For example:
Code: [Select]
jo_file file;
char * temp_pointer;
super_duper_type * awesome_thing;
...

jo_fs_open(&file, "FILE.TXT");
jo_fs_seek_forward(&file, 4096); /* I assume that 4096 is the size of the part to skip in the file and to preserve in the buffer */
jo_fs_read_next_bytes(&file, temp_pointer + 4096, 42);
jo_fs_close(&file);

awesome_thing = (super_duper_type*) temp_pointer;
// or like this: awesome_thing = (void*) temp_pointer;

I think Johannes specified the temp pointer as char because when you use pointer arithmetic like temp_pointer+2 what you really are saying is temp_pointer+(2*sizeof(char));
If you do pointer arithmetic on a int* like int_ptr+2 what you are really saying is int_ptr+(sizeof(int))
If I remember correctly if you try to do pointer arithmetic on a void* it wont work correctly because void has no size, since it has no size the compiler does not know how much to add.
« Last Edit: October 19, 2017, 02:39:57 pm by Danny »

mindslight

  • Administrator
  • Full Member
  • *****
  • Posts: 157
  • Karma: +6/-1
    • View Profile
    • Jo Engine
Re: Reading binary files with pointers instead of returned char stream?
« Reply #11 on: October 19, 2017, 02:53:52 pm »
I think Johannes specified the temp pointer as char because when you use pointer arithmetic like temp_pointer+2 what you really are saying is temp_pointer+(2*sizeof(char));

+1  ;)
I need to read data byte per byte so that's why i'm using a char * (to avoid cast every time)

If you do pointer arithmetic on a int* like int_ptr+2 what you are really saying is int_ptr+(sizeof(int))

Yep :) ptr + 1 gives the same value as ((char *)ptr) + sizeof(*ptr) and also the same values as &ptr[1]

If I remember correctly if you try to do pointer arithmetic on a void* it wont work correctly because void has no size, since it has no size the compiler does not know how much to add.

Nope, the size of a pointer depends of the architecture. (32 bits => 4 bytes, 64 bits => 8 bytes, etc). It's just not portable.





Danny

  • Newbie
  • *
  • Posts: 32
  • Karma: +3/-0
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #12 on: October 19, 2017, 04:14:16 pm »
Nope, the size of a pointer depends of the architecture. (32 bits => 4 bytes, 64 bits => 8 bytes, etc). It's just not portable.
Yes you are correct, I did not explain myself correctly, what I meant is that with void* you don't know the size of what you are pointing to so when you do pointer arithmetic it might not work correctly.
Like you said  the void* size depends on the CPU architecture, all the pointers in an application have the same size and that is why you can simply cast them from one pointer type to another, whatever the data the pointer is pointing to it just contains an address.
You can even have pointers that point to another pointers xD:
Code: [Select]
//have an int
int int_val = 1;
//save its pointer
int* int_pointer = &int_val;
//save the pointer to that pointer
int** pointer_to_int_pointer =&int_pointer ;
//now increase the original int with the pointer to pointer 8D
*(*pointer_to_int_pointer)++;
//int_val will now be 2
« Last Edit: October 19, 2017, 04:20:29 pm by Danny »

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #13 on: October 21, 2017, 02:35:08 am »
Thanks to both of you. It seems to work OK (but I haven't tried on real hardware). Now returning to work on building a kd tree + PVS.

Me and Danny are also starting to look at the SCU DSP. Do you think it can be used for matrix transformation?
I know most people couldn't get anything out of it, but they were using SGL.
The way I see it would be to do matrix transformation on it and DMA the result in the z-sort buffer, but I'm not sure it's possible (with small meshes only, since the SCU DSP RAM is 256x32 bits, which isn't a problem for me)

XL2

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +72/-1
    • View Profile
Re: Reading binary files with pointers instead of returned char stream?
« Reply #14 on: February 20, 2018, 05:41:13 pm »
After doing some tests on real hardware, I came to the conclusion that the read_next_bytes function is super slow.

If I understand correctly, it's reading 1 sector at a time, which causes waits as data gets transfered from the CD buffer to the memory.
As an example, reading a 1,4 Mb file took me around 1 minute 30 seconds, but in a game such as Quake the same file would take around 20 seconds or less to load.

I really like the flexibility with the read next bytes since it allows me to load files larger than 1 MB, but I think it could be improved a lot by doing something more like the asynch read (like read 10 sectors at a time, send the data straight into the specified buffer - Low work RAM in my case -  like in the SBL documentation vol.1 p. 18).

In the meantime, I started to try to write my own functions, but it would be nice to have a speed boost within the default Jo Engine functions.


 

Sitemap 1 2 3 4 5 6 7 8 9 10 
SMF spam blocked by CleanTalk