Indie Game Dev: Automation Servers

Modern programming projects -- commercial ones, anyway -- should use build automation (aka CI/CD). Otherwise there is potential for nasty surprises come QA/release time:

• The person who normally builds a given piece of the final package is away that day and their workstation is locked
• Licenses for certain tools have to be given to everybody in the company who might need to build some artifact, just to avoid the previous problem
• Licenses for certain tools have expired, because nobody's run it in a while and you didn't notice the date had passed
• Which file is the right build? Is it qa-test-build.apk, or release-candidate-June.apk, or 2021-06-20.apk, or one of the myriad copies of each of those in the folder "recent builds", "test builds", "rc builds"? Oh, or did Frank do it this this week?

The right automation software alleviates all those problems and more. It doesn't need a person present in order to complete the correct task, it doesn't forget and it doesn't make mistakes because it's bored or distracted or pressed for time.

If you can only afford one license for a particular build tool, put that license on a single "agent" machine, and everybody can get the finished product from there.

If builds run regularly (you run them regularly, don't you?) then an expired license will make itself known the day it happens, hopefully before the eleventh hour when you're trying to submit a build to the platform.

Build artifacts are organized in a systematic way, so as long as everybody's communicating (Hey Aaron, build #314159 is the release candidate, please test that one), then everybody knows which build is the right build.

Before we get to what I chose, let's talk about how I chose it. I don't have very many deal breakers, other than the ones I've mentioned in the past. Actually, I guess I have two.

First, complicated and/or delicate installation. If the system needs <specific OS version x.y.z.123.b-2>, or <equally specific language runtime>, or has to be installed on a Tuesday evening during a waxing gibbous moon with a chicken in your left hand and a knife in your right, I'm going to pass.

Second, hard to use interface, and especially, hard to find build results/artifacts. If it's hard to set up a job, or hard to find the result of that job and get the build to QA, I may as well build the thing at my desk and email it around to people like an executive would. HARD pass.

Now the must haves. Has to interface with my desired source control systems. Absolutely must understand Subversion and Perforce, and if it doesn't understand Git I probably won't look at it either. "But Ian, I thought you didn't like Git". I don't, but I have friends who provide libraries and some of them keep their source in a Git repo, so I'm better served by a system that understands it.

Has to have a master-worker arrangement. As a mobile guy, I need at least 2 platforms -- OSX for iOS, and Windows or Linux for Android -- just for the application binaries themselves. There might be realtime pvp servers (Linux) or GUI tools (probably Windows) and any number of other things. So I need workers on all of those systems controlled by a master.

Gotta have a web interface. Welcome to the 21st Century.

I've gotten used to being able to "mark a build"; while builds normally get culled after some number of days, or bumped out by the most recent, I should be able to lock one in as a Release Candidate. I should also be able to attach some description to it, so we'll know what that was 2 years from now (rather than just being able to rebuild from a tagged changelist).

I had 3 candiates:

Buildbot - I liked this one because everything is on PyPI so it's pretty much create a virtualenv and pip install to freedom. I had some minor issues at the beginning getting the TLS libraries installed, but I think that's more a function of my inexperience with Python than anything to do with the code.

The master and worker setup are pretty much the same, so once your master is running, you can then install workers wherever you like and point them at the master. It is nearly as easy to do as it is for me to say.

Since it's all Python, I have to assume any place you have a Python install you can have a Buildbot worker. And since it's all self-contained, you can run it manually from your home directory if that's the cut of your jib.

GoCD - a little more professional looking, a little more complicated and a little more fiddly. You can install it via your operating system's package manager, if such a thing exists. Windows has an installer .exe, it's all good.

It's Java, which is mildly upsetting to me, but the installers come with the correct version of it, so I can overlook that just so long as I don't have to upgrade it myself.

My winner was Jenkins, and I'm not gonna lie, it was mostly because I have experience with it. Buildbot seems cool, and if you have to teach yourself one of these systems, it looks like the simplest, i.e. the shortest path. GoCD seems a lot more feature-rich, though the UI seems a little less intuitive, and I find myself having to drill down through the various parts of a job in order to get to the artifacts.

Jenkins is also Java-based, but once again the installation is easy and low-maintenance. I was able to apt install the entire system, including Java, and it wasn't some weird alternate, it was the stock Java that comes with Ubuntu. Sorted.

I've been using Jenkins at work full time for maybe 3 years, and it's pretty hard to find fault with it. It does all the things I've mentioned and a lot more. The UI is pretty good: if you have a bazillion jobs on a single master, and you weren't aware of folders you caveman, you can set up views to organize it all nicely, and that's what we do at work. Some of our leads have created pretty elaborate "landing pages".

Equally as important is that it's very easy to point QA to a specific view if they're testing for you that week. I go as far as Slacking them links to an install page that our build script creates, which is crazy handy. Then they can navigate to said page on their device and click the install link and start breaking my build (and my heart 💔).

I haven't created any of the more complicated job types yet, but I've seen them in action, so I'll be learning how to set them up once the applicable use cases present themselves.

Indie Game Dev: Languages

I wanted to talk about CI/CD systems next, but I'm still evaluating candidates. Maybe next week.

I haven't left very much to the imagination when it comes to me and languages: I've already stated that I'm using Android, and that I view Java as a necessary evil to that end. I like C and Lua. I have plenty to say about plenty of languages -- I've learned somewhere between 30 and 40 of them, depending on what counts (XML? HTML?) and whether variants count (QBasic, Commodore Basic, Visual Basic).

Rather than bore you further with the odyssey of my language scholarship, I'll talk about the few I will be using in this project.

Java

I dislike Java, which sounds a little rich coming from a mobile guy, and believe me, I ain't ever happy about having to use it. That said, most of the Android stuff works with old JDKs (5+ years), and all the tools know how to acquire their own dependencies. So I do the initial JDK install, work through the PTSD, and put it out of my mind.

Don't misunderstand: I don't mind Java as a language, I mind it as an ecosystem. It was created on SPARC workstations running SunOS in the mid 90's, and it behaves as though that's still the only place it runs.

You need something in the ballpark of 10 or 11 environment variables before the compiler will look at your code. One of those is CLASSPATH, and woe be to you if all your Jar files aren't somewhere in there. Seriously, in 25+ years, nobody could update that compiler code to accept some sort of per-project configuration? Everything's defined at the OS level?

So if you're in the middle of a Java project and you buy a new workstation, you spend a few minutes installing Visual Studio, and then the rest of the day preparing your Java environment.

Speaking of defining things at the OS level, why does the compiler care where my files are? If I create a package called org.thenibble.dante and there's a class in it called Worker, then it has to exist in ./org/thenibble/dante/Worker.java. What kind of Elizabethan control-freak bullshit is that?

Python

I have some recent experience with Python, and I'm mostly ok with it, provided we're talking about Python 3. It's been more than a decade, migrate already. I like that Python's ubiquitous, and there's libraries for everything. There's lots of documentation and examples around, and Stack Overflow fills any gaps.

There's a bunch of reasonably good, relatively lightweight REST frameworks, and most of them will work with whatever web server you choose. I'm used to Apache + Flask, but my backend guys at work prefer NginX + flup.

I don't like Python's scoping rules. I don't like that people feel compelled to use classes for everything when it's rarely necessary (though it's hard to tell with the scoping rules). I really don't like the import system, it feels like it's trying to accommodate every guess that every novice programmer ever made, and it's anticipating a few others.

I mean, whitespace?

It's too heavy and slow. Like, unwieldy if you're really churning through a lot of data, which maybe you sometimes do with a game backend. I get that it's scripting and that it's "batteries included", and I'm sure all the correct tradeoffs were made, but sometimes it's just, too, slow.

Lua

I like Lua. It's small and quirky and Really Fast. Don't take my word for it, go hunt down some benchmarks. That's mostly what I like about it, I'm as easy to please as I am to annoy.

There's lots of things to dislike if you want to call me out. It's niche, there isn't a large community or body of libraries. Outside of the obvious things like networking and database and JSON parsing, there's a good chance you'll have to bolt on that third party library yourself.

But that's easy to do because embedding is one of the things it was created for. It's very easy to call Lua functions from C and C functions from Lua. So bolting on that third party library, while a pain in the ass, isn't going to require a systems programmer. (Not gonna lie, having one is better)

Another thing people love to hate: 1-based arrays. Fair enough, it's a bit weird. That said, aren't arrays just for stuffing things into and then pulling out sequentially? Who gives a shit about their offset? If you want to store things and retrieve them from a known location, use tables with string keys. They're still faster than anything outside of C, and the 80's are over, we can spare the cycles.

I wish more people would move to Lua 5.3, it's just better. The previous versions encouraged the use of globals for library tables, and it's just not The Right Thing. I haven't looked into 5.4 yet, maybe it's better still.

C++

I don't much like C++. Sorry, but I don't. We're still writing templates as header-only code because we're still pretending that it's forward-compatible with C, so compilers can't do generics in a sane way. We use the pre-processor instead, like animals, even though the loosest compiler can't compile C code without alterations, because it doesn't allow a boatload of things that C does.

char *ptr = malloc(8);  // 100% legit C, but C++ requires a cast

On top of all that, template syntax looks like line noise; it is truly, spectacularly awful. So you'd better get it right the first time because there's no debugging that shit six months down the road.

Many people are still writing header-only classes for non-template code because evidently we're also still pretending that we can out-optimize compilers by inlining everything. This causes a downward spiral of dependencies, and if I touch any of those headers I have to rebuild my entire codebase because everything includes everything else, and we may as well not have linkers any more. I don't know why, but this is a thing peculiar to C++ people.

I hate that there's no compatibility between C++ compilers. You can't really distribute a pure C++ library, because everybody has their own name-mangling scheme (this might not be true, but it feels true) and their own ideas about linkage and ABI. If you want C++ inside a library, you wrap it in a C interface and you hide the filthy internals from the world.

Finally, any language that plans major changes every three years by decree seems like more work than it's worth. Spending years learning the syntax and nuances of a moving target is for people younger than me. As I mentioned, I already know 30+ languages, I'm just kinda over it.

C

I like C for the same reason I like Lua. It's small and quirky and fast. It took me very little time to learn the entire syntax, and anything I need to do that the base language doesn't supply is in a library somewhere. If it isn't, I don't want it, and I really mean that.

I like that C compilers are actually portable. If I write C code using GCC as my reference compiler, it will build just fine with Visual Studio or Clang, or any other weird compiler you want to dig up. I can also compile these things into a library, shared or static, and I know exactly what the linkage rules are and so will the system that I use that library with five or ten years from now.

What I use in practice is generally a mix of C and C++. I don't mind classes, though I rarely need polymorphism. C's biggest downfall day-to-day is string handling, and there's lots of support in C++, so if I have to do it, I have help.

Those languages really tied the room together

So how is the project using these languages?

• Java for system services, because Android, but only where I have to
• C/C++ via the NDK
• Lua for as much as I can get away with
• Possibly some Python for the backend, I haven't explored all my options with this

Lua is definitely not as fast as C, so once all the logic is in place, I profile any slow parts and convert them to C. This is kind of my normal workflow; it's my slightly-more-modern analog of profiling C code and writing the crucial loops in assembler.

Indie Game Dev: Source control

This probably should have come before Platforms, but I always knew it was going to be an Android game. Source control is usually the first thing I think about, because regardless of anything else you might do, you need to store files somewhere.

If you want to create all your design documents and spreadsheets with cloud tools, I get that. They're generally easy to use, and easy to collaborate with. But if any of those things produce artifacts that go into the build, you need to export them and check em into source control, because that online tool might change or go away.

I need my source control to handle large binary files in a sane way, because who knows what kind of bloated garbage those things are going to generate? While we're on the subject, artists also produce a lot of large binary files, and they too, have to go somewhere.

Everything is distributed/remote these days, so your source control system either needs to be fully network aware, or at least have a feature set that works well over a network. Don't forget, large binary files.

Automated tools will need to check things out, so there needs to be a CLI. Normies -- designers, artists, producers -- need to use it, so there needs to be GUI too, and it should be simple to use.

You need branches, and you probably need to be able to cherry pick changes from those branches from time to time. Maybe you can get away without it, but you'll be a lot less productive, I've seen the before and after.

You need logging and history. When your product actually ships, you'll be fixing bugs and it's easier if you can go through a file's history to see when that bug crept in.

Event hooks are nice. If your system can reject a submission based on a schema or other kind of validation, a lot of errors never make it through the door. You can have notifications, or do backups or any number of interesting things if you know when a specific file changes.

Enough about feature sets. It's decision tree time. What don't I like?

Discontinued/obscure: RCS, CVS, TFS. I especially don't like CVS or anything of that vintage, because it has to contact the server for everything, even just to do a diff. It's inefficient and frustrating, and nobody with any sense is still using any of these things.

Horror of horrors, I don't much like Git, either. The workflow is needlessly complicated, and it uses terms similar to other source control systems, but for different concepts. Sit down in the back, better people than you have tried to talk me into it.

Source control systems that I do like are Subversion and Perforce. Subversion is extremely easy to teach to non-technical folk, and while Perforce is definitely not as easy, its power makes the slightly steeper learning curve worth the investment.

I use Perforce at the office, and it ticks all my boxes. I'm choosing it over Subversion because:

• It's change-centric instead of file-centric
• Cherry picking individual changelists is easy
• Shelving changes is easy
• The GUI, while occasionally... imperfect, has things like revision graphs and "time-lapse view" and I've forgotten what life was like without them.

One thing that irritates me and always will is there seems to be no way to undo changes to a file and keep it checked out. You have to revert the entire file, which removes it from your changelist, and then you have to add it back. If you know of a quick way to do this, please let me know.

It's free to use for up to 5 users, but obtaining the P4V client requires that you hand your email address over to Perforce. Their sales emails are easily ignored.

So there you are. It's Perforce for me.

Indie Game Dev: Platforms

If I were publishing this game, what platform would it be on? Android. Fade to black.

I'm completely serious, but let's see how I arrived there.

I'm just doing this for fun and to build out my pipeline, at least for the first project. So it's not going to be a huge, serious piece. No 3D augmented reality, no social media presence, no infinitely scalable backend hosted out of an old missile silo.

Console is out, they don't give those SDKs to just anybody, and I don't own any consoles regardless. Ouya is dead, and sadly I never bothered getting one. I'm not going to cobble something together with a Raspberry Pi, that's not the project I'm looking for right now.

PC is of course do-able, but it's harder and less fun than you might think. If you're just making a game for yourself that's one thing, but if you have any designs on publishing down the road, you have to face the nightmare that is Other People's Weird Hardware.

I really just want to bang out a quick little game using a simple system. I want something that could be published via Somebody Else's Platform, in case I change my mind and go that route. I'm a mobile guy and I (mostly) understand that space and those systems. So it's gonna be iOS or Android.

Either platform is theoretically fine, I'm equally familiar with both. Let's do pros and cons.

iOS pros:

• C-like native code -- I don't mind Objective C, it's quirky, and I like quirky
• Lots of framework support

I had trouble thinking of any others. Cons:

• Objective C is probably going away in favor of Swift, and I'm too tired to learn another language today, so very tired
• It's a closed system
• You have to have a Mac, and what's more, you have to build and debug with XCode
• You can't build and run an app on your own device without a developer's license
• The license is $100 USD / annum
• The code signing procedure is bonkers complicated, and you have to renew 3 or 4 cryptographic keys/certificates/profiles every year
• IDFA and various other monopolistic practices
• The approval process can be a crapshoot

Android pros:

• Open system
• You can develop and build your apps on any platform -- including OSX if that's your jam
• Once you've built that APK, you can install it on any Android in the world, nobody cares
• A developer's license costs a one-time fee of $25 USD, and you don't need it until you want to distribute the app on Google Play
• You create a single production key once, and signing is all automatic after that. Google will even hold the key for you.
• Advertise with whomever you want. Collect whatever data you want, provided you adhere to the laws of the countries you're publishing in
• Is there an approval process? There is, but I've never seen a rejection. Seriously, never have I ever
• Crash reports that you can symbolicate yourself

Cons:

• Java. *Sigh*. This is mostly mitigated by the NDK, but if you want to interface with any OS services, you're probably doing it with Java
• App permissions are a bit of a moving target

As you can see, I'm no fan of iOS, and the "privacy" shenanigans over the last year have not helped that one little bit. I love Woz, but everything else Apple can be launched directly into the sun.

So... Android. Fade to black.

Indie Game Dev: How I choose software and systems

Most game projects start with an idea, a design, and that seems sensible. But there's nothing better for me than having the whole pipeline worked out so I can start executing on those ideas right away. A big part of this project is having everything exactly the way I want it, so that means I'm building that pipeline.

Where to begin? There are a lot of choices, and they can be paralyzing. I've always started with the deal breakers: the things I absolutely will not do, not use, not put up with. Having some part of your workflow irritate you day after day is not sustainable, so discard those things up front, and keep them out of the ecosystem if you can.

I have lots of deal breakers, so it's easy to whittle the list down quickly. I'll discuss some of them in greater depth in later posts about specific topics.

• Java - it's a necessary evil in places, but if the stock install doesn't satisfy a given system or tool, I'm not going to use it.
• Requiring an IDE to build a resource. Make CLI tools for that, my CI/CD server doesn't have a mouse.
• Not generally a fan of obscure/discontinued tools, languages, or resource formats. The documentation is usually thin, and good luck finding answers on Stack Overflow.
• Also don't much care for Git. You heard me.

Once you've called out all your deal breakers, a swathe of additional tools will be eliminated because they involve those deal breakers. Embrace the constraints, they'll set you free.

With the distasteful stuff out of the way, you can turn your attention to things you really like, and let those things drive your next set of decisions. If you can't have a little fun, why do any of this in the first place?

• I love C and Lua. I like small languages in general.
• I like vscode quite a lot.
• I like JSON.
• I like make.
• I like Jenkins.

If it sounds to you like I've already chosen a lot of things, there may be some, but not necessarily these ones. These are just my preferences, and any real choices have to find their way through this decision tree.

Sometimes I choose things based on what I know best, even if I think something else might be better. Sometimes your hand is forced.

Next time, I'll talk about what I chose first.

Indie Game Dev: How I'd Do It Today

This is my 20th year as a professional programmer. In the early days it was a side hustle -- cobbling shell scripts together to keep the lab ticking over (I got paid, it counts), creating a web page for the company web site, that kinda thing. Eventually I'd "paid my dues" and was able to get the programming jobs I wanted.

(Total bullshit. I did do those things and many more, but I got my first programming job via a referral from a friend. That's how you get the first gig, don't let anybody tell you different.)

I'm a generalist. I've done QA automation, machine vision, network security, web apps, games. Mobile games, probably nothing you've heard of, but lots of them. I'm interested in low level bit twiddling, high level scripting, backend, build engineering, QA, debugging. I've done more time in gaming than anything else, and I continue to surrender to its siren song, but I never really specialized.

A few years back I did indie game dev for real: worked at my own studio, created and released an original title. Our systems guy called it "the best little phone game ...that nowhere near enough folks played", and that feels about right.

I find myself missing those days, or maybe just missing the freedom to do things however the hell I wanted to do them. Even then I wasn't completely free, the studio predated me by a couple of months, so there were a few preexisting systems.

What if I could start out again, do the whole thing exactly the way I wanted? What would I choose? I don't really know either, but I have a plan to figure it out.

I'm going to quietly recreate some portion of the game my studio produced back in 2014. There was a lot of content, so I'm not going to do everything, but what I do make will be top quality, fully functioning, no-sharp-edges software. It will be shippable, but for reasons I won't share here, I'll stop short of shipping it.

This is meant to be the first in a series of posts, and hopefully I can stick to it long enough to convey everything I intended to when I started. What will we be discussing here? Off the top of my head:

• Platforms - what hardware/os will this run on?
• Languages - system, scripting, tools, and more
• Source control - the source has to live somewhere
• Build tools - why is there anything other than make?
• Third party libraries - sometimes you can't roll your own
• CI/CD - because building the shipped product at your desk is The Wrong Thing
• IDEs and editors - vim vs emacs is over
• Content tools - what artifacts do designers create, and how do they do it?
• Live Ops - shipping is now the first step, not the last
• Sunsetting - when the returns diminish, what then?

I'll probably talk a lot about software engineering, because after 20 years I still don't see very much of it, and that makes me sad.

I'm writing this for myself, because we all do, but I do hope others find it useful or interesting. Hopefully I can provoke an "ORLY" once in a while.