OpenGL + CgFX texture/shader drama
So I was implementing the OpenGL renderer, all was going well, until I needed to show a textured quad for testing. I am still using Cg in the runtime, which is not a happy situation for speed, but for now it will do it, in the future I intend to compile with Cg and store the asm into the final production file. So, while testing, the texturing didnt work, and found out that the Cg runtime was requiring the sampler name and not the texture name in the CgFX file, but did worked on the DX9 side. Oh well, just another thing to be aware. I will change it to offline Cg shader processing ASAP, in the NAC (Nytro Asset Compiler).
Component and entity system (and MacOS X)
The engine is using now a component based approach for game entity composition. Also it compiles on MacOS X, but it still needs an OpenGL (ES) renderer.
There is no Scene, World or other major class. There is a SceneRenderer component which is gathering the render packets to be rendered from the entities and their components.
Deleted a lot of code, shrinked and simplified the game manager, easier and faster to use right away to create games.
Worked on the Nytro Project Manager tool, which will basically manage a .project file, where you will specify a game project with all its dependencies (game data folders, the runtime game class name, levels, etc). This manager will be used from within Nytro Studio too (the main game level editor of the engine).
Renaming and first Linux build
Renamed all NYTRO_ macros to N_ , easier to use and read. Plus, had the first successful compile on Kubuntu, the engine was ported to it in a matter of hours (talking about good architecture! 
). Of course no rendering on linux since I dont have the OpenGL renderer done. Only DX9 yet. Once the core will be stabilized, DX11 and GL 4 will come.
Tested the core of the job scheduler, working ok, and having some speed-up on rendering and overall computations. The scene graph and game entities will be threaded, based on dependencies and grouping/affinity.
Multi-core asset compiler and game data file manager
The NAC (Nytro Asset Compiler) is multithreaded, gathers the needed filenames and distributes them on several threads (the number of threads is specified in the NAC config file or command line, and I will probably add autodetection, which will set it from the actual core count of the CPU) . The speed is amazing this way, obviously, I’m really happy with the setup.
The stream manager is now called file manager, easier to find and understand by people, and its based of disk files and zip files (named packages in the engine). The manager has a priority based loading system. When the disk files have the priority, then when you try to load a file, the disk will be searched first, and if not found, the available packages will be scanned. You can add any number of data folders to the engine, also the engine will register automatically any zip files having the .npak extension, named packages, from the data folders.
Removed the EResult enumeration, and replaced all returns with bool. Its easier to read the logic, and frankly I don’t think you need return error code values in the majority of functions, some functions might needed, but should be locally (like for example JSON parser error codes).
Font text rendering and baking
Made the font_baker.exe command line utility to bake font characters into a texture and output a JSON file with the character position and sizes in the texture. It supports unicode character ranges and kerning pairs. Using Freetype for this one.
Font texture (will be packed more tightly if needed in the future by some bin packer algos / atlas packer):
And some test rendering of the font text, observe proper char spacing and kerning:
Asset data compiling and other updates
For the game data, I’ve chosen to have a data source folder and a final data folder. The engine will load binary data, optimized for the given platform. The tools and editor will work with “fat” data, as JSON text files and source data from FBX, PSD. etc.
The NAC (Nytro Asset Compiler) command line utility will take as input the source folder and output the final data folder and it will compile the data. It can skip files not needed to be compiled or it can be forced to compile everything.
The engine runtime loads only optimized and binary data.
I have also made a font baker, which generates a texture with glyphs and a corresponding JSON .font file with info about the glyphs. It supports Unicode characters. The utility uses Freetype to generate the glyphs, so its quite cross platform.
Simplified initialization of the engine, multi-game development in the same engine folder, added project files for a game project which specifies per-game settings, used by game deployment tool and editor/tools.
Other details coming soon!
Refactoring notes
Things are progressing, alot of changes being made right now, hopefully for good puposes.
Here are some changes I am making to the engine:
- get rid of STL for runtime and use custom containers and string classes (faster compile, more control over the container/string features)
- use strings only where needed (faster code, less memory(or mempool) fragmentation)
- use IDs instead of names, for various objects for the runtime (faster checks/lookups and data loading)
- separate subsytems so they will not use pointers, but handles for the objects (better for making them as multi-core jobs)
- use #pragma once instead of old school .H guards in headers (the header looks cleaner and pragma once is supported by the major compilers)
- split files with multiple classes in separate files (faster compilation)
- putting the variable system to good use, for settings, runtime tweaking and so on (aka console variables)
- using alot of forwarding and avoiding including too many headers in one header
- not using precompiled headers (promotes cleaner headers)
- shortening all the names to reasonable lengths (Ex: getGraphicsProvider()->getCamera() to gfx()->camera() )
And many other things I will post in the next posts. As always, stay tuned..
Engine major refactoring (aka.. the road to perfection is cursed)
I know, the dreaded “refactoring” word, but its part of the learning process, plus it makes things exciting.
So, the engine will be refactored in such a way that you would not need STL, and even strings in the engine’s code, I will use a hashed id system for resources, data, as few pointers as possible, and no parsing or expensive dynamical things at runtime, assets will be always cooked to highly optimized production formats specific to each platform, easy to load and feed to the job system.
I will have to revamp some old container code to replace STL, imho STL is good for offline processing in tools, when speed is not such a critical factor and it helps faster development of tools and source data file formats.
Since major systems will be rewritten, I will try to keep the blog updated with the changes I make.
This “revelation” was started by some DICE’s presentations and some other internets reads like http://BitSquid.se .
In the past the engine was not emphasizing on speed too much, but I think its time I go into optimizations more, a good reason to learn new things also. Stay tuned for some architectural decisions blogging.
Editor, cutting edge rendering and multi-threading
While attending GDCE 2011, I went to some talks about multi threading engines, and I thought I should split and multithread the engine’s subsystems, making them independent data-wise.
Another thing is high quality rendering, brought to extremes, no legacy support for lower end machines, DX11+, target rendering quality: (CGI movie) http://www.youtube.com/watch?v=VCTtIIW1WA0
Editor side, command based editor, empty at first, then UI filled up by plugins. As for the GUI SDK, Qt will be used.
Also trying to clean up the code, to release an alpha SDK for people to play, its not written in stone, but lets hope I will do it.
Cross platform and of course cross compiling
With the grow of Android and iPhone, I had to keep up also, so the engine is now relatively easy to compile using CMake project files generator, on Windows, Linux, MacOS, iOS, Android and maybe XBox (but I do not own a XDK, yet), no PS3 because is a hell to develop for..
Cleaned up the engine, secured code with some more const-ness, and researched Qt for editor UI, so far I’m pleased by the functionality and free licensing, so I will probably make the editor in Qt. The editor functionality is separated from UI, so it will not be a full editor rewrite, just the UI itself.
Renamed some longer file extensions, and stripped out the DevIL image library from the engine, now the engine only loads a custom texture format named .NTX, which contains images slices (1 slice if a single texture, or 6 if cubemap) and their mipmaps. The format will be documented. This move unloads some size from the final nytro.dll/so, makes it smaller.
Also by using this new building system, there is also full static libraries, so you can link engine+game+plugins in one single exe.
Another idea is to make the editor be usable only with one hand a the mouse 
, having alot of gizmos inside the scene will help that, and less toolbar buttons.
News and screenshots coming soon, thanks for all who watch this blog, it keep me walking further with this!
