The State Machine

Gameplay in Steel Circus decomposes into just a handful of basic states:

• You have the ball
• Your team mate has the ball
• An enemy has the ball
• Nobody has the ball (it’s either lying on the ground, or it’s in flight after a shot)

Exit conditions for these states are also trivial:

The exit condition for the first three is that whoever had the ball now doesn’t have the ball anymore (i.e. they shot it, or it was taken from them). The exit condition for the “nobody has the ball” state is that the ball now has an owner.

If a state exits, the state it switches to is also almost entirely governed by the same set of conditions, namely “does the ball now have an owner and who is it?”

The way I set this up is to have each state check if its exit condition is met, and if so, ask the state machine to set its next state to a “Default State”. This Default State is simply a hub that immediately exits to the appropriate next state depending on who now owns the ball. You can think of it as similar to the “Any” state in Unity’s Mecanim.

The Ball Ownership State

There are three, mostly independent actions that the AI needs to take care of when it has the ball:

• Throwing the ball, either passing or shooting goals
• Moving
• Dodging

Throwing: Evaluating Confidence Levels

The AI constantly evaluates pass and goal shot opportunities while it has the ball. For both, it samples various aim directions and calculates “confidence” levels that throwing the ball in a given direction will result in either a successful pass or goal shot.

For goal shots, calculating confidence means taking several vectors towards points between the enemy goal posts and figuring out if a clear shot is possible, and how likely it is that an enemy will intercept that shot (based on the enemy’s position and their potential movement radius, given the time that the ball will travel to get to its destination). If confidence for a clear shot is low (because there are enemies or bumpers in the way), the AI also samples pseudo-random shots towards nearby walls and evaluates their confidence level, giving it the ability to score goals by bouncing the ball off the wall.

Evaluating confidence of a successful shot means taking into account the chance that an enemy will intercept that pass or goal shot, which is complicated by the fact that the ball can bounce off of walls or bumpers. We handle this by splitting the path of the ball into several segments and evaluating the chance that a team member or opponent will intercept the ball at each slice.

Evaluating pass confidence is only marginally more complicated: we sample directions to pseudo random points around team mates and evaluate both their chances of intercepting the ball and the opponents’ chances. If the AI can’t come up with a good direct pass, it will again sample wall shots to see if it can find one with a higher confidence. 

On top of that, we take a few extra steps to tweak the AI towards desirable pass behaviour:

• We evaluate each team member’s position, using the same position evaluation settings that the Ball Ownership State uses for movement (we’ll go into the details later on). We don’t evaluate passes to team members that are in a significantly worse position than ourselves.
• The bots should favour passing to human players, so on any but the highest difficulty settings, we penalize passing to other bots by scaling down the calculated confidence levels.
• Back-passes often come as a surprise to human players, so we reduce confidence levels for these.
• Passes around the own team’s goal are also heavily penalized, and if the AI calculates that the ball would hit its own goal if a pass fails, it discards the sampled shot direction altogether.

Throwing: Threat Levels and When To Throw

Every tick, the ball owner AI determines a best pass direction, a best goal shot direction (if applicable) and confidence scores for these two shots. As the main purpose of the AI is to support human players and we don’t want it to play too well, we only let it make shots when a few conditions are met:

• It has held the ball for some minimum amount of time.
• Its confidence that a given shot will work is higher than some threshold value.
• Any potential pass target isn’t in a considerable worse position than the AI itself.

None of these conditions are limited by hard numbers – instead the thresholds are given by curves dependent on other factors:

• All three of the above conditions have different thresholds, depending on the perceived threat that the AI is under. Each tick, the ball owner state calculates a perceived threat level, depending on the proximity of enemy players and whether or not they can currently tackle the AI. The higher the threat level, the lower the minimum confidence to take shot, minimum position score of a pass target and minimum time the AI needs to hold on to the ball after becoming the ball owner.
• As time goes by, the minimum confidence for shots, as well as minimum position score for team mates also goes down. We want the AI to pass the ball back to human players, instead of carrying it across the whole field, unless it absolutely has to.
• The AI strongly favors making passes over shooting goals. It needs to be pretty certain that a goal shot will land. There’s little fun in watching the AI win the game for you, so the bots need to be team players, not rock stars.

Moving: Details on Influence Maps

The entire positioning for ball ownership is done with influence maps. Again, you can think of an influence map like a heatmap – every few ticks, the AI samples several potential positions around itself and assigns them scores based on several gradients that represent desirable or undesirable target positions. If the AI is under elevated threat (i.e. opponents are close), it will take samples more often. The gradients for ball ownership are:

• Proximity to opponents: heavy penalty if close, medium penalty at mid-range.
• Proximity to team mates: slight penalty – if all things are equal, we would rather spread out.
• Proximity to bumpers: we don’t have dedicated code to make the AI steer around bumpers (and yeah, it shows), but we penalize target positions close to them.
• Proximity to the playfield’s top and bottom edges: close to the edges of the field, you have less options to maneuver, so we penalize those slightly.
• Proximity to own goal: this incurs a medium penalty, but over a large distance – we want the ball owner to move away from their own goal quickly.
• Proximity to enemy goal: this is rewarded heavily, particularly at close range. We want the AI to be able to try and carry the ball into the goal, even when there are enemies nearby.
• Close proximity to the playfield’s left and right edges: near the own goal, there’s a heavy penalty. This just serves as an additional influence to make the AI try and get the ball away from their own goal as soon as possible. However, we also penalize positions along the enemy’s edge of the playing field – this serves to make the AI not get stuck around the enemy’s goal corner, but seek out positions where they get a clear goal shot.
• Horizontal position: the farther into the enemy’s half of the field, the higher the position score.
• And finally, we reward positions with clear pass lines to team mates. This isn’t a playfield-based gradient, but a function that takes into account current team mate positions and checks nearby enemies and bumpers to see if there’s a clear pass.

One way to implement an influence map is to update an entire grid of positions at regular intervals (i.e. create an actual map). Despite the visualization screenshots in this article, we’re not actually doing that – instead our states produce lists of pseudo-random potential target positions around the AI’s position and evaluate these individually. In the case of the ball owner state, 25 samples every 15 ticks or so (higher intervals if the AI is under threat) are sufficient to make the AI carry the ball past enemies and bumpers and into the enemy goal.

Another interesting aspect about the ball ownership state: Once we evaluated our list of prospective target positions, we take the top three and actually evaluate several points along the route to each of them. If you are the ball owner, you care as much about enemies or obstacles en route to your destination as you do about the score of the destination itself.

Dodging

Dodging used to play a vastly different role in Steel Circus: When you had the ball, you could do a dodge which made you temporarily invincible to enemy tackles (though not other skills). This had major implications for when and where the AI would dodge, and it factored in its ability to dodge (i.e. whether or not the skill was on cooldown) into its threat evaluation when it had the ball. Since the last update, dodge is just a quick change of position that uses up stamina and lets the player do abrupt movement changes, but doesn’t grant it any sort of immunity against oncoming enemies.

The AI now uses dodge more like a disruptive behaviour to make its movement less predictable: when it gets the ball, it determines a random delay after which it will execute a dodge if enemies are continuously nearby. Once the delay is over, it will use the position evaluator to sample random positions around itself (at distances equal to the dodge movement distance) and execute a dodge in the best direction. Depending on the difficulty settings, it also suppresses throwing the ball during the dodge and for a short duration afterwards.

Part 1: Overview
Part 2: The FSM and Ball Ownership
Part 3: Other States
Part 4: Optimization and Future Work