Behind the Scenes: porting NifSkope

DigitalBox
Dev
December 22, 2024

Today's article will be a long one, on the theme "Behind the Scenes".

We will review how the port of the NifSkope software was done to Haiku.

It will be the opportunity as well to show how to use HaikuPorts to do the job, but also to see all the pitfalls encountered.

Porting is not easy

Let's be honest : porting a software to another system is not an easy task.

But if you have enough knowledge on the corresponding software and patience, it can be feasible depending on the complexity of the application.

Today, let's review how the port of NifSkope to Haiku was done !

If you don't know what it is, NifSkope is a tool for opening and editing the NetImmerse file format (NIF).

NIF is used by video games such as Morrowind, Oblivion, Skyrim, Fallout 3, Fallout: New Vegas, Civilization IV, Dark Age of Camelot...

In the process of porting a software, the tasks below are generally needed :

Know how to compile the software on another system (Linux, Mac, Win)
- Pay attention to build specificities
- Understand if the build instructions are outdated or not
- Understand the build tool used (cmake, make, qmake, ...)
Identify the dependencies needed by the software
Understand if some configuration files are required
Be prepared to patch some files
Know how to debug
Be able to create a pull request (PR)
Ask for help to the community if needed
Be patient :)

Compiling on Debian

Ok, as explained before. let's first compile NifSkope to another system : it will be Debian.

The below command will install Qt5 on Debian :

sudo apt install qtbase5-dev qt5-qmake qtbase5-dev-tools

Then the NifSkope github repo is cloned and the build process is launched :

git clone --recursive https://github.com/niftools/nifskope cd nifskope qmake make

First issue and bad news, the compilation is not working on Debian due to a missing include :

If we check at the build instructions on the website : they are outdated and the latest release of the software on that repo was done in 2017...

You can also notice that the "--recursive" option was used for cloning the repo on github.

It will allow to retrieve the sources of gli, qhull and zlib libraries directly, and this part will add complexity when writing the port for HaikuPorts :

Ok let's try to compile NifSkope again :

make

Great this time it's working fine with the QAction include added.

Let's check the dependencies :

cd release ldd ./NifSkope

You can notice here that NifSkope is depending on several libQt5* libraries but also on libGL.

Identifying the dependencies

One important step before compiling is to be able to identify the libraries dependencies.

As reviewed before on Debian, the program depends at least on :

Qt5 libraries
libGL
libz

If you can identify these dependencies before compiling, it's better.

However there's no magic process to do so, especially if it's not clearly indicated in the build instructions, so be prepared to take some time during the compilation process below, if you have not identified them completely.

For NifSkope, the below have been identified :

In case you need additional libraries for your port, you can check among the ones available on Haiku :

search -a | grep _dev

Compiling on Haiku

Ok now let's do the same but for Haiku.

If you didn't compile NifSkope before on another system like Debian, you wouldn't have noticed that :

Qt5 was required instead of Qt6
The QAction error was there, and was not relative to Haiku

That's why the advise "know how to compile the software on another system" is important.

On my side, I've already compiled NifSkope on MacOS several times, that's why I was more confortable to identify the pitfalls.

Let's install Qt5 dev tools on Haiku :

pkgman install qt5_devel qt5_tools Then clone the repo : git clone --recursive https://github.com/niftools/nifskope cd nifskope

And modify the "lightingwidget.cpp" :

src/ui/widgets/lightingwidget.cpp &

As explained before, this issue is a general issue, and for MacOS, I have also patched this file as visible on https://github.com/DigitalBox98/nifskope/commit/2e4d9beb2dd951ebe3670c78a970ceff373e3b5a

Let's build NifSkope :

make

After a few minutes I can see :

Great !

This a really an important step when working on a port : having a first compilation completed.

Depending on the complexity, it can take hours or days before being able to compile because :

the builds instructions might be outdated (qt5 was not indicated clearly for instance and the QAction include was missing)
maybe you didn't read carefully the build instructions (like missing the "--recursive" option which is a must have, I already made that mistake...)
dependencies need to be identified and installed

So far, the good news is that the first compilation is behind us !

The bad news is that we are only at the start of the process :)

Executing NifSkope

Let's go into the "release" folder :

cd release ls

And execute the program :

Like the message said : Oh no!

At that time, I didn't know how to do some debugging on Haiku, and the issue was really hard to identify.

When trying to reproduce the process, I would suggest to go into the debugging land by generating an executable with debugging info :

cd ..
make clean
qmake CONFIG+=debug
make

If you want to know how to setup QtCreator for debugging, please read the article "Debug with GDB".

Launching the program again in debugging mode, we start at the main :

And when running, the crash is at the "setViewport( ogl )" level :

It takes maybe more than a month of delay before resolving this. It's because I was doing other things meanwhile and the resolution was by chance :

Posting the error on the Haiku Forums, someone gives me some ideas and I have decided to move the instruction a bit later in the code (after the initMenus()):

Then building again the software :

qmake make

And launch it :

cd release ./NifSkope

Yes, yes, yes !

The program is displaying some warning messages -we will resolve later-, but at least it can render NIF models like below!

As this runtime error has only been noticed on Haiku, I suspect there would be something to fix in the corresponding library (libQt5OpenGL maybe). But at least, a workaround was found.

Having the first compilation done is a first step, but having a working binary is another important step !

At that time, I was like a child, having a new toy and proud to have made my first port on Haiku. But it's not yet finished for sure !

HaikuPorts

Now the tricky part : HaikuPorts.

How does HaikuPorts work ?

It expects the below :

a recipe file : indicating dependencies and how to build the software
a patchset when needed : to apply some patches
an URI of the sources : the golden source to retrieve the software's sources (indicated in the recipe)

The general process for HaikuPorts is to first download the sources from the URI, then apply the patchset if indicated, compile the software, and finally proceed with the creation of the package ("hpkg" file).

If you need to use HaikuPorts for your port, you can follow the article "Build with HaikuPorts" for setting up your environment.

HaikuPorts has some similarities with SPKSRC, which is another tool I already used to port some applications for NAS synology via cross-compilation.

First recipe

Ok let's start by creating a first recipe.

Checking the haikuports folder :
cd haikuports ls

My first step is to identify the folder/category for NifSkope : it will be "media-gfx".

Next is to synchronize with the latest updates on the repo and to create a dedicated "nifskope" branch :

git pull git checkout -b nifskope git status

As NifSkope is now available on HaikuPorts, I will remove the existing folder and create a new empty one:

cd media-gfx

mv nifskope /boot/home/Desktop/nifskope-saved

mkdir nifskope

cd nifskope

Then I need to create a first basic recipe :

lpe nifskope-2.0.dev9.recipe

The name is important here and it must follow the naming convention : <package_name>-<version>.recipe.

Let's fill the recipe with :

General information
Dependencies
Build
Install

if you want to reproduce the process, do a copy/paste of the below content in Pe editor : 

SUMMARY="Tool for opening and editing NIF files"
DESCRIPTION="NifSkope is a tool for opening and editing the NetImmerse file format (NIF). \
NIF is used by video games such as Morrowind, Oblivion, Skyrim, Fallout 3/NV/4/76, Starfield, \
Civilization IV, and more."
HOMEPAGE="http://www.niftools.org/"
COPYRIGHT="2005-2014 NIF File Format Library and Tools"
LICENSE="GNU GPL v3"
REVISION="1"
SOURCE_URI="https://github.com/hexabits/nifskope/archive/refs/tags/v2.0.dev9.tar.gz"
CHECKSUM_SHA256="zzz"

ARCHITECTURES="?all !x86_gcc2 x86_64"
SECONDARY_ARCHITECTURES="!x86"

PROVIDES="
   nifskope = $portVersion
 app:NifSkope = $portVersion
 "
REQUIRES="
 haiku
   "

BUILD_REQUIRES="
   haiku${secondaryArchSuffix}_devel
   "
BUILD_PREREQUIRES="
    cmd:g++
 cmd:make
    cmd:qmake
   "

BUILD()
{
   qmake
   make $jobArgs
}

INSTALL()
{
    make install

    mkdir -p $appsDir
   cp $sourceDir/release/NifSkope $appsDir/

    addAppDeskbarSymlink $appsDir/NifSkope
}

 You can notice that :

SOURCE_URI : it indicates the link to the sources archive to download for NifSkope
There's no version "2.0.dev9" indicated : I think it's deduced from the recipe filename
LICENSE : the license type is required (I will change it later)
BUILD_REQUIRES : it describes the build dependencies, I will complete it later
BUILD_PREREQUIRES : it details the build tools required (qmake + make + g++)
BUILD : required to explain how to build (qmake + make here)
INSTALL : required to explain how to install the HPKG (it copies the NifSkope executable to the applications folder + create a deskbar link)

Please note the SOURCE_URI is not referring to https://github.com/niftools/nifskope, because the latest version distributed is available at https://github.com/hexabits/nifskope which is a fork of the original repo :

Let's check the compilation via HaikuPorts :

hp nifskope

Fix the SHA-256 checksum for the sources download URI in the recipe by the below :

SHA-256 : b40baca5fc3b11292cd284b5ef0d07dbe40c3bab8067c05b7f25bdb6141fe26c

Ok now let's compile again :

HaikuPorts: compilation errors

The error is not straightforward here, but I can notice the below :

WARNING: Failure to find: srg/gl/BSMesh.cpp WARNING: Failure to find: srg/gl/BSMesh.h WARNING: Unable to generate output for: /sources/nifskope-2.0.dev9/Makefile [TEMPLATE vcapp]

Looking at the "Nifskope.pro" file in the "work-2.0.dev9/sources/nifskope-2.0.dev9/" folder, I can notice some strange things :

Why, why, but why the developer has released a source archive which is having a typo issue ???

It has been fixed on day later, but never packaged in a release :/

The other point which I'm already aware - because it was the case also for MacOS port : I need to remplace TEMPLATE vcapp by TEMPLATE app.

So, yes, I will need to do my first patchset.

First Patchset

Let's fix the "NifSkope.pro" file :

cd work-2.0.dev9/sources/nifskope-2.0.dev9/ lpe NifSkope.pro &

And let's create the patchset :

git add NifSkope.pro git commit

I then hit Ctrl+O and Ctrl+X to make the changes effective.
The "git commit" is required to make the changes effective for the upcoming patchset.

** Important ** : I need to go back to "nifskope" root dir before creating the patchset (else it will not work, trust me!):


cd ../../..
hp -e nifskope

Let's check what is the patchset content :

lpe patches/nifskope-2.0.dev9.patchset &

Ok now I need to add that patchset in the recipe via the "PATCHES" keyword :

lpe nifskope-2.0.dev9.recipe &

I can continue the build, but now I will redirect the errors to a log for clarity :
hp nifskope 2>nifskope_compile.log

If I check for the details, I can see the below :

Do you remember the "identify the dependencies needed by the software" ?

Looking at the "NifSkope.pro", these dependencies are described below :

This is about gli and qhull libraries.

The gli_devel library is not available via pkgman, and qhull_devel is available but unfortunately NifSkope's project requires "c" files in addition to headers files (left=qhull_devel from pkgman / right = qhull sources for NifSkope build) :

When I check in the NifSkope "lib" folder, I can see the below :

On the left is what I have currently via my HaikuPorts recipe versus what is expected by NifSkope build on the right : I can see that "gli" and "qhull" folders are missing.

Crap ! :-)

This is due to the "recursive" option proposed by the git tool which is making the port to HaikuPorts a bit tricky to do.

How to resolve that ?

The idea is that I will download two additional source archives via HaikuPorts:

the gli source archive
the qhull source archive

Let's add the below lines in the recipe :

SOURCE_URI_2="https://github.com/g-truc/gli/archive/refs/tags/0.8.2.0.tar.gz" CHECKSUM_SHA256_2="9e7024c2df77c011eff4f66667c1834620c70b7902cd50f32ab48edd49fe0139" SOURCE_URI_3="https://github.com/qhull/qhull/archive/refs/tags/v8.1-alpha1.tar.gz" CHECKSUM_SHA256_3="09e5e4c5b2b8a9e617a46876fef5a3d33e70aa1d08a163ff05d37701327c3be7"

And once the archives are extracted, the BUILD must put them into the "$sourceDir/src" folder :
cp -r $sourceDir2/gli-0.8.2.0/gli/* $sourceDir/src/ cp -r $sourceDir2/gli-0.8.2.0/external/glm/* $sourceDir/src/ cp -r $sourceDir3/qhull-8.1-alpha1/src/libqhull $sourceDir/src/

This part was also a bit tricky, because as you can see (to make things easier), the "gli" sources include a "glm" part which need to be copied to the root "src" !

Ok, now let's check the results :
hp nifskope 2>nifskope_compile.log

Some errors are still there :

As the dependencies errors are not all shown in one time, I have to run the build several times to identify the below dependencies to add in the "BUILD_REQUIRES" :
devel:glm devel:libGLU devel:libqhull_r devel:libz

Let's build again:
hp nifskope 2>nifskope_compile.log

Why ? But why NifSkope is unable to locate the zlib.h whereas it has been added in the dependencies ?

The reason is quite simple :

"zlib.h" is available in the root headers directory, whereas the project is expecting "zlib/zlib.h". So I need to patch the "bsa.cpp" file :

cd work-2.0.dev9/sources/nifskope-2.0.dev9 lpe lib/fsengine/bsa.cpp &

And I generate the corresponding patchset with the below command :
git add lib/fsengine/bsa.cpp git commit cd ../../.. hp -e nifskope

Let's verify the patch :
cat patches/nifskope-2.0.dev9.patchset

I can continue the build :
hp nifskope 2>nifskope_compile.log

HaikuPorts : linkage errors

The next error is a good news :

All the compilation phase has been completed and now the linker (ld) is trying to build the executable by linking all the various parts :)

I can check the error details :

This is relative to zlib unable to be linked with NifSkope.

The method I found to resolve this issue was to add the zlib as a parameter for the link phase via the LIBS variable in the NifSkope.pro project file.

cd work-2.0.dev9/sources/nifskope-2.0.dev9

lpe NifSkope.pro &

LIBS += -lz

I had to do the patchset again :
git add NifSkope.pro git commit cd ../../.. hp -e nifskope

Now I'm confident, launching the build again :
hp nifskope

HaikuPorts : first executable

Ok there are still some errors, however the binary has been built with HaikuPorts !

Bingo !

Let's launch it to check.
cd work-2.0.dev9/sources/nifskope-2.0.dev9/release ./NifSkope

You know what ?
It's not a problem : I already had this issue before :)

Let's patch the "nifskope.cpp" file :
cd ../src lpe nifskope.cpp &

And let's move the setViewport() line just below the initMenu() :

Let's do another patchset :
git add nifskope.cpp git commit cd ../../../.. hp -e nifksope

And verify it :
cat patches/nifskope-2.0.dev9.patchset

Let's build it again :
hp nifskope

Ok now I can launch NifSkope - which should not raise the blocking error :

cd work-2.0.dev9/sources/nifskope-2.0.dev9/release ./NifSkope

Yes, I have a first working executable with HaikuPorts !

It means that so far via HaikuPorts :

I have downloaded all the required sources for the application
The patchset was applied accordingly
The compilation is working fine
The linkage phase is ok
And I have a minimal version running !

Is the job completed ?

No. The last error I had prevent to build the HPKG package :

I know that NifSkope need two configuration files :

nif.xml
kfm.xml

So let's add them.

As far as I remember, to make things harder, these xml files are not available directly on the github repo of the project.

So I had to download each file via SOURCE_URI_4 and SOURCE_URI_5 below in the recipe :
SOURCE_URI_4="https://github.com/niftools/nifxml/archive/refs/tags/v0.9.0.tar.gz" CHECKSUM_SHA256_4="984c247115bc49f90ded69b4c8feb9a62a0365ad53f5890312302b6a05bb4ed7" SOURCE_URI_5="https://raw.githubusercontent.com/niftools/kfmxml/develop/kfm.xml#noarchive" CHECKSUM_SHA256_5="3af5634dfef643494bb7646924b1b18ab22bc0d9b5ea4758d4e6b693c78a2f3e"

Then I had to copy these files during the BUILD process:
mkdir -p $sourceDir/build/docsys/nifxml/ cp $sourceDir4/nifxml-0.9.0/nif.xml $sourceDir/build/docsys/nifxml/ mkdir -p $sourceDir/build/docsys/kfmxml/ cp $sourceDir5/kfm.xml $sourceDir/build/docsys/kfmxml/

To complete the setup, I had to define these files has system settings file :
GLOBAL_WRITABLE_FILES=" settings/nifskope/kfm.xml keep-old settings/nifskope/nif.xml keep-old "

I know it's not the best approach, but for the moment it's the only one that worked for me.

As these settings files will be in the settings/nifskope directory, I have to copy them during the INSTALL phase:
mkdir -p $settingsDir/nifskope/

cp $sourceDir/release/nif.xml $settingsDir/nifskope/ cp $sourceDir/release/kfm.xml $settingsDir/nifskope/

It means I need also to patch the files which are using these configuration files to match the new folder :

find . -name kfmxml.cpp lpe src/xml/kfmxml.cpp &

Replacing "kfm.xml" by "/boot/system/settings/nifskope/kfm.xml" will do the job :

I know that's it's not the best patch, as it can be improved to indicate to change it only in case we are compiling under Haiku.

But as I'm using HaikuPorts, it means I am compiling under Haiku:)

Let's do the same for the other configuration file :

lpe src/xml/nifxml.cpp &

Replacing "nif.xml" by "/boot/system/settings/nifskope/nif.xml" will do the job :

Ok now let's complete this by a patchset and rebuild (note: I had to apply one additional build before the patchset command, ie before "hp -e nifskope") :

git add src/xml/kfmxml.cpp git add src/xml/nifxml.cpp git commit cd ../../.. hp nifskope hp -e nifskope hp nifskope

If I look at the result I can see :

You know what ?

The first time I've tried to package NifSkope, I didn't know about these blue warning messages.

This is due to missing package dependencies as the "REQUIRES" section is empty for the moment :

These blue warnings are a nice help here, because instead of identifying them manually, let's put the missing list indicated!

HaikuPorts : first package built

Now I can build again :

Oh my God !

It's amaaaaazing ! (with the accent ^^)

Yes the first package of my NifSkope port with HaikuPorts has been built :)

Let's check it :

open ../../packages

And launch it :

Let's open it :

Really great !

No error message anymore at the start of the application as the configuration files are correctly retrieved and this display of a NIF model is fine :

NifSkope port : improvements

I noticed that compiling NifSkope with g++ lead to a big issue : the rendering window was resizing randomly in a crazy way.

When the compilation was done with clang++, this weird behavior disappeared.

Let's fix that :

cd work-2.0.dev9/sources/nifskope-2.0.dev9/ lpe NifSkope.pro &

I indicate to qmake to use for C/C++ the clang/clang++ tools :
QMAKE_CC = clang QMAKE_CXX = clang++

Then I create the corresponding patchset :

git add NifSkope.pro git commit hp -e nifskope

Let's add the below line for "clang++" in the PREREQUIRES section in the recipe :
cmd:clang++ >= 18

And let's build NifSkope from scratch :

rm -rf work-2.0.dev9/ hp niskope

I confirm that this time, the NifSkope port is really working fine :)

Now let's complete the license part.

NifSkope has a specific license, that's why I've done a copy/paste of its content in the file named "NIFSKOPE".

Then at the recipe level, I've created the below directory :

mkdir licenses

And I've put the license file :

In the recipe, I have then indicated the license filename :

NifSkope port : icon

Now let's polish the port by adding a nice icon in the INSTALL section of the recipe.

I had to update the recipe by adding the below lines inside the INSTALL section :

How about the icon file ?

If you want, you can download the "hvif" icon for NifSkope on the HaikuDepot NifSkope page.

Then click on the link named "Download icon in 'hvif' format" :

In my situation I have created this icon in Icon-O-Matic manually (it was not an easy task:) ):

The next step is to export the icon into "HVIF Rdef" format

Then I had to copy it inside the "additional-files" folder at the same level than the recipe file:

mkdir additional-files cp /boot/home/Downloads/nifskope.rdef additional-files/

Looking at the content of the file, it needs some adjustments :

The below header must be copied inside this file :

resource app_flags B_SINGLE_LAUNCH; resource app_version { major = @MAJOR@, middle = @MIDDLE@, minor = @MINOR@, variety = B_APPV_FINAL, internal = 0, short_info = "@APP_NAME@", long_info = "@LONG_INFO@" }; resource app_signature "@APP_SIGNATURE@"; resource vector_icon {

Like below :

Exporting it to "nifskope.rdef.in" makes the job :

I then removed the original file, as it's no more useful :

rm additional-files/nifskope.rdef

Now I had to indicate in the recipe to recognize that file :

That's it !

A final build is necessary to check the last changes :
hp nifskope

Let's check the results once the package is available :

The license file is there. Now let's launch the application :

Great the icon is now attached to the app !

Moving forward

Once I was happy with the results, I have then submitted a Pull Request (PR) of my "nifskope" branch into the github HaikuPorts repo.

If you have read this article until the end, well done, I hope you have enjoyed it, because it's not an easy one :)

If you would like to understand more about HaikuPorts recipes, you can check the below pages :