Jo Engine Forum

Sega Saturn Development => Project announcement => Topic started by: SkimmingSanshiro on April 25, 2018, 03:04:19 pm

Title: Virtua Skimmer
Post by: SkimmingSanshiro on April 25, 2018, 03:04:19 pm
Heres my project i've been working on. Its a Tony Hawk style Skimboard game, you do whats called a "wrap" on a wave and do as many tricks as possible without falling 
(https://camo.githubusercontent.com/566f15f0fc546af6be43a92dc9b3527ae9665f7c/687474703a2f2f692e6d616761696d672e6e65742f696d672f333536392e706e67)

Its still a work in progress but I reached the dynamic memory usage limit and probably be awhile before I can do a significant update since i'm still learning basics. Recommended emulator is Yabause as SSF is too fast. On Saturn hardware everything is a bit to fast(the wave especially) but probably more the speed im looking for finished. The water dithered mesh that moves rapidly im going to replace with a Grandia water style raster effect as the current jerking motion is almost flashing colors at you.

Issues-
runs about %50 to fast on sff %40 to slow in Yabause compared to hardware(Overall speed in general not finished)
Out of dynamic memory for more transformations
Printf tricks dont clear when finish run
Gameplay is about %30 (No real collision/Hacky Gravity + Controls)

To Do-
Move mesh/Textures off main ram to free up dynamic memory
Grandia Style Water effect(VDP2 Raster effect?https://youtu.be/Foln_F9ggk8?t=1m46s (https://youtu.be/Foln_F9ggk8?t=1m46s))
Link system (Tricks must be done quickly together or lose link)
Animation
Convert title screen meshes to hardcoded images
Collision/Physics
2 Player
Fishing mode using saturn mouse

https://github.com/VirtuaSkimmer/Virtua-Skimmer-Sega-Saturn-
https://www.youtube.com/watch?v=yqetF9Q5tG0
Title: Re: Virtua Skimmer
Post by: XL2 on April 26, 2018, 01:58:09 am
 Good job!
I'll try to give you a hand if I can find some free time.

Using the tool I made you should fix all your memory issues.
You probably allocate memory for sprites but don't clear it.
Also make sure your sprites are loaded from the disk, not hardcoded.

For the framerate, when you use "slSynch ()", it will wait unitl the end of the allocated time before rendering again, so it shouldn't be too fast nor too slow depending on your settings. The jo core run also calls slSynch at the end of each loop.
You can try different settings (like your speed increases x units * nb of frames).
Make sure your emulator also has its frame limiter option on.

For the vdp2 effect you want, it involves using line scrolling.
I never tried it yet myself, but I think there is one SGL demo using it.

For collisions, you can simply use a bounding box or sphere and use the quads' normals to calculate where it should push you.

You will need a basic spatial subdivision for that (a grid is the easiest choice and what I would suggest you to try).
Title: Re: Virtua Skimmer
Post by: XL2 on May 14, 2018, 04:16:10 am
Just to help you a bit with your project.

I made a model converter and binary loader, so you can use it to load your player and other entities.
It allows you to load multiple models and display them easily.
It should fix your memory issues if you use the binary files (ZTP).
It doesn't read the texture coordinates, so you need to either flip your quads yourself or flip the textures (a bit of a pain, but that's how the Saturn works sadly, with no texture coordinates).

For flat shaded quads, use a 8x1 texture, as my converter doesn't read the MTL file. I plan to transform these 8x1 textures to normal polygons (untextured) at a later time.
Make sure your images have no more than 16 colors and are saved as 32 bits TGA images (no RLE compression).

I might switch to FBX to allow animations and I'd like to add realtime gouraud shading someday, so for now it's just simple and dumb lightning, but anyway enjoy!

NB : The model converter is in the TOOL subfolder. The ZTI image converter is also there.

Title: Re: Virtua Skimmer
Post by: XL2 on May 14, 2018, 11:30:21 pm
Ok, small update, I added realtime gouraud shading.

The per-vertex normals might not be 100% right, so the results might not be as smooth as what you get with the official Sega 3d tool.

Enjoy!

EDIT : I think I got the per-vertex normal OK, see the demo in newer posts.

Title: Re: Virtua Skimmer
Post by: ponut64 on May 14, 2018, 11:44:08 pm
Impressive stuff, XL2. Thanks immensely for sharing.
That takes a 53 KB .h format file down to 16KB.
Title: Re: Virtua Skimmer
Post by: XL2 on May 14, 2018, 11:53:17 pm
In fact, the .h file takes the same amount of memory (it's in ASCII form, so it takes more memory, but after compiling, it's all binary).
But the most important thing is that it's not hardcoded, so you can load different models for different levels and all.
Title: Re: Virtua Skimmer
Post by: ponut64 on May 15, 2018, 12:56:12 am
That makes sense, not something I have explored enough to understand. When I get that far though a tool like that should be invaluable.
And that explains why you'd need a function to draw the polygon since normally the .h files include that.
Title: Re: Virtua Skimmer
Post by: XL2 on May 15, 2018, 02:54:42 am
And you also have to consider that the textures aren't included in the h file.

That being said, I just made a small update, I think I got the per-vertex normals right as the whole shading seems smoother now.
I did the update in less than 10 minutes, but it seems to work well.

I also added the vertex normals in the .h file, but I didn't try to see if it works.

I updated both the tool and the demo.

Title: Re: Virtua Skimmer
Post by: XL2 on May 16, 2018, 04:00:22 am
I tested it on real hardware, I tried with 7 models (around 180 quads per model) and the results are rather good, with 60 fps with 4 models, then the VDP1 can keep up, but the CPU doesn't. By not updating each frame, it might be possible to keep 60.

https://youtu.be/XapzA2yPEAU (https://youtu.be/XapzA2yPEAU)
Title: Re: Virtua Skimmer
Post by: SkimmingSanshiro on May 16, 2018, 06:22:43 pm
Love the gourad lighting and this tool. One thing though when I try adding my working ZTP loading into my project it wont build when I add the void computeLight() and void display_model(entity_t * model, bool UseRealtimeGouraud)

Ive added everything else and successfully builds including the gameloop() and removing Jo engine callbacks and run(). I didn't load the ZTP to the current address or anything in my_draw() yet only exception. But when I add those 2 voids it wont build. I replaced the ZT folder with the new one and the makefiles match
Title: Re: Virtua Skimmer
Post by: XL2 on May 16, 2018, 06:28:47 pm
It might work with the Jo Engine callbacks, I'm just not sure because I think it does things with the gouraud shading. Best way to know is to try it.

As for why it doesn't work with your project, I don't know, make sure all the addresses are right.
Also make sure that your are using subfolders and clean up your CD folder as you are limited to the number of files per folder (just remove all those TGA textures, you don't need them anymore).
Remove all the unused header files too to save memory.

If all fails, just do it the other way around : rebuild your game around the demo I gave you.
Title: Re: Virtua Skimmer
Post by: ponut64 on May 17, 2018, 05:01:57 pm
Well, I did get realtime gouraud working, but only with a header file. The important thing is XPDATA, which isn't explained in any of the English documents I have (only PDATA, which obviously does not do real time gouraud). Which requires face normals, which jo engine converter does not make (understandably). That makes your tools necessary, and thank you for that.

I did try to get the binary files working. I can get them to compile no issues, but the model loading routine clearly conflicts with jo engine. (Screen opens to a glitched screen reminiscent of loading the file, then fades to black, then ceases) I'd have to rebuild based on your system routines. Making the Z-Treme engine truly an engine of its own being incompatible with the core of Jo.

I suppose if I wanted to build an extensive project, I (or rather we) really needed those binary load-on-demand files. I'll spend some more time experimenting later on what exactly I will need to put in and take out for it to load properly.

Another thing about your binary files is that they seem to require textures to display. Do you intend on it working with texture-less / material-less models? Not too important, as one could just slap a material on everything and put a white texture on it. Haven't made a conclusion there yet, again, need to do more experiments.

Title: Re: Virtua Skimmer
Post by: XL2 on May 17, 2018, 05:44:17 pm
For the textures, you can just put a 8x1 single color texture. The tool doesn't read the material data sadly as I never bothered trying to read the mtl files, but it will still convert it as a single color polygon, so it really only takes 4 bytes + 32 bytes for the color data (always 16 unsigned shorts). It was the easiest way to get the tool out in a reasonable amount of time.

The Jo Engine convert does in fact face normals, it's the per vertex normals that it doesn't calculate. I will try to add functions to precalculate lightning for static lights, mainly for the maps. It should help make everything look much better while also being faster than dynamic light. I still need to do some reading on gouraud shading.

For the loading routine, load your Jo Engine specific data first, reinit the CD with my own function, then use the ZT functions.
Binary files are the way to go, but there are restrictions with folders and filenames (max 8 caracters plus 4 for the extension).

Since I wrote so many tools for my game and used many functions differently (like audio), it didn't make much sense to init everything just to reinit almost everything right after, so I'm not using Jo Engine anymore for Sonic Z-Treme, but I will still try to contribute to improve it.
Title: Re: Virtua Skimmer
Post by: ponut64 on May 17, 2018, 08:37:13 pm
I got it working, my friend!

After adding ztCDinit,
I had to (in one stroke so I do not know for sure which one of these was the culprit for the freeze) comment out the text prints and the fade-in / fade-outs as well as the clear screen as well as slScrAutoDisp (in the model loading function).

I have to run now but my model still isn't displaying but Sonic is. I'll get that figured out, just a matter of exporting it all right.
Title: Re: Virtua Skimmer
Post by: corvusd on May 17, 2018, 08:57:51 pm
I tested it on real hardware, I tried with 7 models (around 180 quads per model) and the results are rather good, with 60 fps with 4 models, then the VDP1 can keep up, but the CPU doesn't. By not updating each frame, it might be possible to keep 60.

https://youtu.be/XapzA2yPEAU (https://youtu.be/XapzA2yPEAU)

Very interesting implementation XL2. You are using one light Dynamic source for all models. 1200 quads lighted whit source. Really nice. You think if is possible improve the performance or use of the CPUs? More Lights and whit color? Thanks! :)
Title: Re: Virtua Skimmer
Post by: XL2 on May 17, 2018, 09:35:28 pm
You can easily put one light source per model, but not more.
SGL can only register one source.
The way around this would be to not use SGL, or maybe it's possible to use offsets, but I don't know.
I might compute lights offline for static lights, which should allow much better colors.
Maybe you could then use the SCU DSP to calculate the additions/substraction on top of static lights, but I think it would be slow.

I already tried with colors, it's super easy.
For this demo, I'm using "semi white", but you can put any color you want.

It's already using the 2 CPUs. As soon as you call slPutPolygon, it will look to see if the slave is busy, and if not it will transfer the command so that the slave handles everything.

Title: Re: Virtua Skimmer
Post by: ponut64 on May 18, 2018, 12:04:32 am
I am just going to comment on how immensely forward-thinking Sega was with how intelligently ... multi-threaded? SGL was for the mid-90's.
I was keen to insist on the gouraud shading being done on the slave since I want the master free for my undoubtedly immensely sloppy game code, but I guess Sega was looking after lesser experienced 3rd party devs like me after all.
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 12:08:25 am
Yes, you don't need to worry too much about the slave CPU, you can use it for maybe one function as the main CPU retrieves controls and stuff like that, but otherwise it's mostly for 3D rendering.
But they didn't make use of the SCU DSP at all!
Most devs agreed that it was great for lightning calculations
Title: Re: Virtua Skimmer
Post by: SkimmingSanshiro on May 18, 2018, 06:27:02 am
So im getting along pretty well porting my game over the your engine. Turns out taking all the .h files out solved the problem. sltranslate doesn't seem to translate the mesh anywhere though? For example

slTranslate(100, 20, 50);

Is the same as any other combination

Code: [Select]
slPushMatrix();
    {
        slTranslate(100, 20, 50);
        slRotX(sonic_rot_x );
slRotY(sonic_rot_y );
slRotZ(sonic_rot_z );
        computeLight();

        display_model(&entities[0], 1);
       

    }
    slPopMatrix();
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 11:57:16 am
You forgot to do "toFIXED". You can also do something like (150 <<16). The Jo Engine function does it. But I recommand you only use fixed values for more precision (I'm not sure if the newer Jo engine versions -which uses fixed values - has sin/cos tables for them - if it does, you can use it also).
Title: Re: Virtua Skimmer
Post by: ponut64 on May 18, 2018, 04:39:12 pm
At some point a new thread in the "shared code" section should be made about this converter. Though I see there's a bit more activity on segaxtreme than there is here.

Regardless, how do you export your OBJs? I can use the model converter on your Sonic file errorless, even changing the textures (with new ones), and it converts and displays properly.
If I use my own model, it converts fine (no errors) but it does not display. I ask about how you export your OBJs (or handle the models?) because I can import your OBJ file into Blender and then re-export it, at which point if I use the model converter on it again (using either my settings or Blender default), it does not display. Using my demo or yours, and your demo polycount correctly reports with the non-displayed models that there's no polygons happening.

The textures DO load, to the best of my knowledge, as the converter gives information about them (size 8x1, depth around 2800, RLE 2) and states they are written into the file and your demo pauses and states loading textures from that. Must be about my OBJ file.

I'm using Blender 2.78a.

/e: One last thing. Any chance of loading the binary files from a BIN folder on the CD? Just to keep things neat. At present it seems they can't be anywhere but the root dir.
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 05:03:09 pm
Works fine here. Maybe your model was too big so what you saw was actualy inside it, so it did backface culling and you didn't see anything?
I just scaled it down 10% in Blender and then scaled it up 4 in the tool.

The only other thing it could be is that you use RLE compression, so it didn't actualy read the textures.
I'm also using Blender, so it shouldn't be an issue.
I just use the default values, see attached image.

About the subfolders, you can add the following functions to your code and call them (set the directory, then after set it back to root).
I suggest you first call ztCDsetDir, load everything you need to load, then call ztCDsetRoot to return one folder back.
 

Code: [Select]

void    ztCDsetDir(char * subDir)
{
    Sint32  fid;
    if (subDir != NULL)
    {
        fid = GFS_NameToId((Sint8 *)subDir);
        GFS_LoadDir(fid, &gfsDirTbl);
        GFS_SetDir(&gfsDirTbl);
    }

}

void    ztCDsetRoot()
{
    Sint32  fid;
    fid = GFS_NameToId((Sint8 *)"..");
    GFS_LoadDir(fid, &gfsDirTbl);
    GFS_SetDir(&gfsDirTbl);
}

Title: Re: Virtua Skimmer
Post by: ponut64 on May 18, 2018, 05:41:50 pm
It works now, and it would seem the only thing I had to do differently was to make a subfolder in the IN folder for the converter.
/e: Actually, I also put the file name in all-caps.

I don't know, my dude, that is the only thing I did differently. Thanks for sanity-checking my mesh!
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 05:44:09 pm
I just noticed Blender supports exporting animations.

I will play a bit more with that in the next few weeks and see what I can do with it.
Title: Re: Virtua Skimmer
Post by: ponut64 on May 18, 2018, 05:49:10 pm
What that does is it exports each frame as a separate mesh according to the range of frames you put in the timeline and the desired frame-rate you put in the Blender render section ("Dimensions" tab).

I don't think you need an example of how that can be organized into a file but here it is. As you suggested, its converted header files and only the host mesh has complete mesh data. Each other frame has all but its vertices axed and structured as new meshes.
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 05:57:41 pm
I know next to nothing about animations in Blender, but if you can send me your model with some basic animations (like 2-3 different movements) I could see what I can do.
But doing vertex animation isn't cheap memory-wise, so it might be better to somehow compress the data.
I really don't know how I would handle it.
Maybe by going with 4 bytes per vertex instead of 12, but I'll have to think about it.
Title: Re: Virtua Skimmer
Post by: ponut64 on May 18, 2018, 06:40:13 pm
Here you go.

My rig is terrible but this should serve well enough for examples.
If you want, I can poke around and try and get a blend file from someone more experienced. (Of course, you might guess by now what subject material it will be ...) These really are my first Blender / 3D animations.

Target 15 FPS (you can change that of course)
0 - 7 Start Walk
7 - 15 Walk Part 1
15 - 22 Walk Part 2
22 - 29 Stop Walk

29 - 34 Mime Part 1
34 - 36 Mime Part 2
36 - 44 End Mime

If it really is only 12KB per frame in memory, that's enough for me, so I understand if you feel its more important to work on something else.
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 06:48:06 pm
Thanks, I'll look it up.
I won't have time this week, but maybe I can get some work done next week.

I think I will just ask the user to enter : how many animations, how many frames per animation.
So, if you say : 4 (nb of animations), it will ask you how many frames for animation 1 (let's say 7), 7 for animation 2, etc.
And it would store the default vertices (your default obj file) and for each animation frame 4 bytes per vertices to store the offset.
It should be cheaper than storing all the vertices multiple times, but it would still be a little bit hard on the CPU.
Anyway, we'll see once we get there!

If you manage to make it work with your current implementation, make sure to show us with a small clip!
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 07:53:47 pm
Now that I think about it, it might not even work because of the normals.
SGL does backface culling depending on the normals, not the vertices AKAIK.

Can you try it to confirm it does indeed work?
The work around would be to also compress the normals, but that would double the amount of memory!
The other way to do would be to animate using meshes as bones, it would require less memory but it's still not that easy to implement since I will have to interpolate the movements.
Title: Re: Virtua Skimmer
Post by: ponut64 on May 18, 2018, 07:57:31 pm
I had it working using header files as given, however I didn't circum-rotate the mesh much I just saw it working in a few perspectives.
It may be I just didn't move the mesh enough to see anything go wrong. This wasn't with the real-time gourad shading.

When you do have time, take a test of the header files in my previous post (specifically pony.h). Still needs XPDATA for every frame rather than PDATA ( jo engine mesh data ) but that shouldn't take long to do, just only change vertices data.
/e: Now I realize my test was without XPDATA, so I guess I wasn't really testing what we were talking about. With PDATA it did work.
I guess later today I am going to take a closer look at how it works with XPDATA / real time shading.
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 08:01:22 pm
Would you mind showing me (if you can just do a short private youtube video or something as I'm not on my computer now)?
If the planes are single plane, I think they will get culled for more intense rotations, and that's a big problem.
Title: Re: Virtua Skimmer
Post by: ponut64 on May 18, 2018, 09:05:08 pm
https://youtu.be/M00oA1ZEKU0

This may not be conclusive but it was easy to do.

You know what, I'll just try and throw in something that really should break it if it works like we think it does. Give me a minute.

/e: Yep, as you expected, things get culled when they would be otherwised blocked in the models' rest position.

https://i.imgur.com/2JassMX.png?1

I'd guess some way of compressing the normals is important

There is probably a really good reason why the animations in Quake are like they are.
Hopefully we can at least prevent writing redundant textures.
Title: Re: Virtua Skimmer
Post by: corvusd on May 18, 2018, 09:30:41 pm
You can easily put one light source per model, but not more.
SGL can only register one source.
The way around this would be to not use SGL, or maybe it's possible to use offsets, but I don't know.
I might compute lights offline for static lights, which should allow much better colors.
Maybe you could then use the SCU DSP to calculate the additions/substraction on top of static lights, but I think it would be slow.

I already tried with colors, it's super easy.
For this demo, I'm using "semi white", but you can put any color you want.

It's already using the 2 CPUs. As soon as you call slPutPolygon, it will look to see if the slave is busy, and if not it will transfer the command so that the slave handles everything.

I know for you, the easy way right now is use the SGL. For 1 Light Dynamic the SGL is very well implemented.

But my questions was for this data extract from https://segaretro.org/Sega_Saturn/Technical_specifications#Graphics (https://segaretro.org/Sega_Saturn/Technical_specifications#Graphics), for the theoretical maximum capacity for SS in this type of process:

DSP geometry processing: 188 MIPS (million instructions per second)

    Fixed-point operations: 114 MOPS (million operations per second)
    Additions: 85 million adds/sec
    Multiplications: 85 million multiplies/sec
    16-bit divisions: 5 million divides/sec

Geometry calculations: 114 MOPS fixed-point calculations

    Vertex transformations: 2,400,000 vertices/sec
    Polygon transformations: 1,800,000 polygons/sec
    T&L flat lighting: 800,000 polygons/sec
    T&L Gouraud lighting: 700,000 polygons/sec

Polygon rendering performance: Lighting

    800,000 polygons/s: Flat shading, 32-pixel polygons
    500,000 polygons/s: Flat shading, 50-pixel polygons
    200,000 polygons/s: Gouraud shading, 32-pixel polygons

Texture mapping performance: Lighting

    300,000 polygons/s: 32-texel textures
    200,000 polygons/s: 70-texel textures
    140,000 polygons/s: Gouraud shading, 32-texel textures
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 09:43:19 pm
The main issue with SCU DSP is that you need to DMA everything, wait for the dsp to complete the operation and then stop it, DMA the data back.
It really requires perfect syncronization else you just waste time waiting or DMAing little data.
Plus it's in assembly only...
Title: Re: Virtua Skimmer
Post by: XL2 on May 18, 2018, 09:47:24 pm
https://youtu.be/M00oA1ZEKU0

This may not be conclusive but it was easy to do.

You know what, I'll just try and throw in something that really should break it if it works like we think it does. Give me a minute.

/e: Yep, as you expected, things get culled when they would be otherwised blocked in the models' rest position.

https://i.imgur.com/2JassMX.png?1

I'd guess some way of compressing the normals is important

There is probably a really good reason why the animations in Quake are like they are.
Hopefully we can at least prevent writing redundant textures.

Textures are only stored once in memory using my tool. Plis they are 4 bits per pixel, so  you really don't need to be worried as I'm sure you have a ton of VRAM left.
Even with Quake maps and 1000 textures I still have some VRAM left.
For the normals, you could also just use dual_plane for now, but that's not an option for me.
I will try to read more on the subject.

EDIT : Just looked at your video, it still looks quite smooth! If you don't use that many polygons, you can probably just use dual plane quads and keep the current implementation.