Suunto Spartan Ultra Trail Test

Original Review date: August 2017. Tested with the original HR Strap and 1.6.14 – 1.9.36 firmware

The model tested here is the Suunto Spartan Ultra Titanium and was acquired through normal retail channels.

Summary

Coming off the back of the top scoring Ambit3 we had high hopes for Suunto’s Spartan Ultra. You’d think putting Ultra in the name and that eye watering price point is surely indicative of something special for us trail and ultra types.

But after +5000km on the wrist we’re finding it hard to summarise the Spartan. Is it like a fēnix 3 with better accuracy (it outperformed the fēnix 3 in all tests) but fewer features? Or an Ambit3 that’s less accurate with different and less useful running/outdoor features, but with a colour screen? Or something else again?

It’s very accurate distance-wise in easy GPS conditions but pretty mediocre in more difficult conditions typical of NZ trail running, yet has supurb elevation climb accuracy. It also has an excellent screen, but limited in its screen customisation, and lacks a number of key trail/ultra running features.

While it is improving in both accuracy and features with recent firmware releases, as an offering to trail and ultra runners the 1.9.36 FW version could still be summed up as ‘incoherent’. On the basis of current testing we couldn’t recommend it as an upgrade or a first purchase. The equation of accuracy + features = price just doesn’t stack up when compared to the other options available. Though it is feasible that could change with future FW releases, just don’t bank on it.

Note: We are yet to test the endurance aspects of the Spartan Ultra so are unable to comment on battery life and battery saving modes.

A recent model round-up can be seen here, and the testing methodology is here

What kind of trail runner is it good for?

Maybe those with deep pockets but are scared of going into the woods? With excellent elevation performance and distance accuracy in easy GPS condition, could be good for an open sky big vert runner? It lacks a flexible feature set for highly structured interval training nor does it offer virtual racer/pacing, so mostly urban serious racers will be let down too.

What should you expect as a long term running companion?

A fair weather friend. When the going is easy it’ll be reliable and look good on your wrist. If that’s all you need then it’s a perfect match.

Good Stuff

An excellent screen, sharp and visible in all conditions. Group-topping elevation/climb accuracy, and near top distance accuracy in easy conditions. A great navigation screen that distinguishes where you’ve been from where you’re headed as well as any waypoints en route. Elevation profile of planned route. Vibrate alerts (finally!). A magnetic charging clip that is super easy to use. Buttons are much more responsive than the Ambit3.

Bad Stuff

Screen rather less visible after you’ve smeared it with mud using the touch screen. Weird limitations around screen customisations by activity. Poor all round positional accuracy and substandard distance accuracy in typically difficult (ie. NZ trail) conditions. It really doesn’t like tree cover. No pacing/virtual racer functions and limited interval work outs. No storm alerts, no off-course navigation alerts, no waypoint approach alerts, no pace/HR limit alerts, no custom timer alerts (eg. walk-run). No distance or ETA on planned courses. A standard foot pod can’t be used to completely override the flakey under canopy GPS performance (it autocalibrates from the GPS). No local PC access to your running data, it is available to you only at the pleasure of Suunto.

Plea to Suunto

Please continue the Ambit series for serious trail and ultra runners. The approach of releasing an unfinished product with promises of improvement is not a welcome one, please only release watches when they are actually finished and fit for purpose. The ability to fully customise the watch activity screens should be a given. Alerts? Where are the alerts (nav/pace/HR/timer/etc)? And its about time proper pacing made it to the feature list. The Movescount mobile app still doesn’t have the ability to see recorded laps. And course distance remaining/ETA. Then give us the tools to manage our running data as we like – not simply via a few selected partner services.

Formal Testing

Accuracy

So far Spartan Ultra testing has covered over 320km on the surveyed courses over 38 separate days with total of 146 comparison laps. Outing conditions ranged from fair weather bliss to full thunderstorms flash-flood fun (we run in all conditions, just like you). Our testing is designed to provide a statistically robust measures in real world conditions – check our methodology here.

So far we’ve tested the Spartan Ultra using the ‘Best’ GPS setting with GLONASS on and off. Testing has so far included four firmware versions (1.6.14 to 1.9.36).

The GPS peers against which the Spartan Ultra was tested included the following models: Suunto’s Ambit2 & Ambit3 Peak, Polar’s V800, Garmin’s FR910XT, fēnix 2 & fēnix 3, and a Sony Xperia Z5c with the SportTracker app.

During testing multiple GPS devices are carried as a means of validation, if all units have a bad GPS day then a tin foil hat is donned. Though some days with poor satellite availability saw all units a little degraded (some more than others), no testing days saw GPS chaos across all models.

Note: Aside from the initial overall figure, which tells us if will record long or short over the long term, we report distance accuracy using absolute percentages. In other words 5% wrong is 5% wrong, it doesn’t matter if it’s long or short, they don’t cancel each other out during your run.

Distance

The Spartan Ultra has a split personality on the distance accuracy front. Overall  it registered 98.0% of the total distance, meaning it’d short you around 2% in your long-term training log if you run in conditions similar to ours.

In easy to mixed GPS conditions its performance is excellent, coming in at 98.6% and 98.5%  respectively. By comparison the test group averages were 98.3% and 98.3% and the best performer came in at 99.1% and 98.8% (yes the margins are tight).

Where its personality switches is in difficult GPS conditions where the Spartan Ultra recorded 95.4%, just under the group average of 97.3%. This mediocre performance in difficult conditions actually pulls the overall average of the Spartan Ultra at 97.7% down to just above the group average (97.3%).

The impact of GLONASS is ambiguous, while there is 1.5% improvement in accuracy when running in difficult conditions, easy and mixed condition performance is actually marginally down on the GLONASS off setting. The figures where GLONASS is on are: 98.4% (easy), 98.2% (mixed), 96.6% (difficult), 98.0% (combined).

At this point you might be thinking those margins are pretty small, practically the same even. But there’s another way to look at accuracy where we call each recorded distance within 1% of the truth as a success. Think of it this way – you’ve just run a 10km PB and you want your glory on Strava or the online service of your choice. But your watch shows 9.9km at the finish, so no digital PB, and no bragging rights for you. So even a 1% error can be a bummer, let alone 4%.

Our ‘within 1% success test’ is much more sensitive in identifying how reliable each recorded distance run is. The differences between models now becomes a lot more pronounced. Here the Spartan Ultra recorded within 1% of the true distance 31.7% of the time in easy conditions compared with 58.5% for the best performer and 27.9% for the group average. This plummets for all models under difficult conditions to 3.1% for the Spartan Ultra, 8.7% for the best performer, and 4.5% for the group. In all, at 21.8% across combined conditions the Spartan is still above the group average at 19.8%.

On the un-surveyed and truly difficult course consisting of gorges and ridges under a big tree canopy (fairly typical in NZ) the Spartan Ultra came in well short of pretty much all other models. While the true distance of the course is unknown – we really ought to make a day of surveying it – the Spartan recorded distance was 7% under current estimates (which themselves are likely short). Though again this was run with GLONASS off, we’ll retest with it on sometime.

So what gives

When we plug all the factors we record into a model to show us which are associated with distance accuracy we can see the Spartan is strongly effected by satellite availability (well a general proxy of it), tree cover, and mildly effected by track sinuosity (bendiness). On the other hand the raw GPX feed accuracy is predictive of good distance performance. Put simply Spartan Ultra accuracy is inevitably going to suffer in bush trails and when satellite availability is not great.

Positional

Unlike distance accuracy, the positional offset is down on the group average across all conditions. In fact under easy GPS conditions it is one of, if not the worst we’ve tested with a median offset of 2.4 metres (the best performer here was 1.4m). It fares a bit better in mixed to difficult GPS conditions only coming in 2nd or 3rd from the bottom of the group at 2.6m for both, still almost double that of the best performer. Needless to say it comes in worse off than the group average.

And just like distance accuracy switching on GLONASS improves the difficult conditions performance a bit. But it’s not a clean sweep and it doesn’t leapfrog the group average as a result.

Unsurprisingly then, when the Spartan Ultra was taken into genuinely difficult conditions (heavy tree cover, steep terrain, gorges), it had trouble getting reliable positional fixes. The repeat lap tracks looked like a 4X MTB race, all tracks go in the same general direction but have phases of wild slaloming. The Spartan Ultra’s median offset distance between tracks was bottom of the group at 11.4m compared with 3.9m for the best performer and 5.0m for the group. Note: this was FW1.6.14 with GLONASS off, we’ll retest and given an update sometime.

Note: positional accuracy is measured as GPS trackpoint to the closest point of a surveyed track (or repeat track where course is un-surveyed), so figures reported here will be a lot better than the actual positional accuracy.

Satellite vs Watch Distance

And for the nerdier types among you, here’s how the watch recorded distance lines up with it’s raw satellite distance. The charts also show how each and every recorded lap distance pans out over the testing.

The recorded distance you see on the watch is in purple whereas the yellow is from the unfiltered gpx track. The vertical lines are the averages of the two. Our general performance expectation is for the filtered recorded distance to be a tighter distribution that pulls the average distance closer to 100%. And indeed this is what happens with the best performing watch we’ve tested.

Unfortunately here Suunto’s special sauce filtering appears to drag the watch recorded distance down while the distribution is scarcely improved. You can also see the deterioration of lap accuracy in the difficult GPS conditions.

Track Appearance

Tracks under reasonable GPS conditions generally appeared pretty responsive and sharp to changes in direction. That said the recorded track itself is often not centered on the actual path followed and can be seen shadowing the surveyed course by a few metres. This kind of makes sense of the discrepancy between recorded distance and positional accuracy in the Spartan Ultra.

The trackpoint cloud image below shows the density of GPS position updates from recorded runs against part of surveyed course. The relatively open sky view conditions (tracks to the right) appear relatively well centered over the track though still widely dispersed. The limited conditions (track on left) shows rather wider and less focused tracking with track shadowing evident.

The darker the colour the better focused the tracks are. The grey line is the actual course.

Elevation

We’ve run well over 5,000 metres of vertical on the LIDAR surveyed courses with the Spartan Ultra, and when it comes to elevation, the Spartan Ultra clears out it’s peers.  In terms of overall accuracy it comes in on top at 94% of the total climb compared to 85% for the group. Similarly the Spartan Ultra came in within 10% of the actual metres climbed 75% of the time, well above the group average of 51%.

Elevation range, even if we didn’t calibrate the watch, was generally good. And where it was calibrated before the run, the elevation range and points where true elevation is known were excellent.

Note: Like distance we calculate elevation in terms of absolute percentages (10% off could be 10% high or 10% low). The survey course probably is a bit tough on GPS watches with respect to elevation with lots of rapid 10-30 metre climbs/descents. We are guessing watches would probably perform better on longer more pronounced climbs/descents.

Battery Saving Data Recording

Sorry, we haven’t ventured here yet. As well as changing the sampling rate, battery savings can also be made via colour resolution and display time-out settings. We’ll get on it when we can.

Functional Tests

Data Syncing and Sharing

So Suunto have gone all the way with the Spartan Ultra with syncing requiring direct contact with Suunto servers both for PC and mobile. No internet, no sync. This also means that your data goes directly from the watch to the Movescount site where you can access it. There is no meaningful data on your PC/mobile so all data sharing has to be by the grace of Suunto’s Movescount service.

We’ve commented before that the shift of data away from the user (owner) to a proprietary service is an unwelcome development. While having brand agnostic aggregating services like Strava makes this a little more palatable, there really needs to be a means of bulk managing/downloading the original data from Movescount.

While there are options to get other GPS brands (Garmin/Polar) history into Movescount via MXActivityMover. Getting your history out of Movescount doesn’t appear quite so easy. At this point it feels we’re losing ownership of our running data, and making it more difficult to switch brands or use non-Suunto endorsed run analysis services.

Mobile syncing is done through the same Movescount app as the Ambit series uses which is pretty stable these days (on Sony Xperia Z5c running Android 7.1.1). PC syncing requires the Suuntolink program, rather than the old Moveslink. This all works well enough and in 2017 we haven’t endured frequent the server outages of years past.

Navigation Functions

Route creation is done via the Movescount site, the same as the Ambits. The OpenStreetMap based topo maps make route creation straightforward and includes a rough elevation profile and the option to add waypoints to it. It’s also easy to import a GPX to the site to create a course.

When navigating the route we can see the elevation profile and the navigation screen is great. We can now distinguish between where we’ve been and where we are going (breadcrumb vs blue line) and zoom as required. Distinguishing these two is really helpful when navigating complex courses with double up sections and multiway junctions.

Unfortunately, unlike the Ambit3, there are no alerts when we approach a waypoint or wander off course. So that’s one step forward and one back – or maybe two back. And disappointingly there is still no distance remaining, distance to next waypoint, or any ETA estimates with the route navigation.

In addition to the route navigation there’s the waypoint navigation where you can navigate via compass bearings and distance to any waypoints you’ve saved. Sadly we still are having to manually add waypoints either via the Movescount site or on the watch itself as there are still no waypoint management tools available.

Finally there’s a general breadcrumb trail screen which can be used to traceback your aborted adventure to the start point. There’s also a findback navigation function which provides bearing and distance back to the start of the activity.

Pacing Functions

This bit is easy, there aren’t any. None. This is again a step backwards from the Ambit series which at least had pacing functionality via the custom apps functions. And the Ambit’s themselves weren’t so good on the pacing side, lacking any virtual racer function (ie. racing against your past recorded events). Incredible that Suunto continues to ignore this when every other brand has this great training and racing feature.

Battery Run Down

Sorry, we’re waiting for my herniated disc to sort itself out before we attempt any 20 hour trail run-downs.

Standard Feature Set

General Trail Running

• GPS accuracy under canopy : Nope
• Consistent GPS accuracy : Yes in easy GPS conditions only
• Rapid GPS Acquisition : Yes
• HRM : Chest belt
• Cadence option : Wrist and foot pod (no manual calibration)
• Battery 8hr with HRM and high accuracy setting : Yet to test
• Barometer : Yes
• Breadcrumb with waypoint navigation : Sort of
• Vibration alerts : Yes
• Trail legible display: Yes
• Open data access : Nope
• Fully ANT+ or BLE Compliant : Yes

Ultra Feature Set (as per trail running plus)

• Battery 14hr+ with HRM and high accuracy recording : Yet to test
• Battery 24hr+ with HRM and down-sampling : Yet to test
• Electronic compass : Yes

Nice to Have Features

• Mobile uploads : Yes
• HRV (R-R) recording with recovery estimate/test : Yes
• Footpod GPS override : No
• Basic interval workout ability : Yes
• Pacing function : No
• Position/waypoint autolapping : No
• Custom timer reminders : No
• Everyday watch : Yes
  • Activity Tracking : Yes
  • Mobile Notifications : Yes

Non-Tested Core Features

While functions like heart rate and cadence are core, either we’ve got no means of objectively testing them.

Heart rate appears like any other belt based monitoring we’ve seen in recent models (if you wash your strap frequently enough that is). Similarly the wrist based cadence readings seems good enough for our general purposes. Worth noting if you are running with poles any wrist based cadence is worthless.

While the Spartan Ultra accepts standard bluetooth foot pods there is no means to manually calibrate it to give pace and distance independent of the GPS. The manual suggests it is periodically calibrated via the GPS which defeats the purpose for us. The reason we’d be pairing a foot pod in the first place is to bypass GPS based data. Will still be interesting to see if the foot pod makes a difference in low battery saving GPS mode – we’ll let you know.

Yep, like any other sports watch these days the Spartan Ultra has a bunch of daily activity tracking and recovery measures. No sleep monitoring though, and no inactivity alerts to tell your lazy arse to get out of that chair.

General Useability

As a daily watch, the Spartan Ultra is really very nice. The smart notification functions work well with the crisp screen. But we are not sold on the touch screen as a feature. During daily use it works well enough, though results in constant smears. When you are out on the trail mud and sweat present much bigger issues. We struggle to see any benefit of a touch interface for a small screen when your fine motor skills are the last thing you want to be exercising. Give us decent buttons.

On the topic of the buttons, they are an improvement over the laggy ones of the Ambit3. The audible alerts (not that there are many of those on the Spartan) are loud enough though the vibrate is a little soft when you are going hard out.

Anyway, back to the screen, it’s a big improvement over the Ambit series and the  activity layouts are highly in legible all conditions except maybe for the smallest screen fonts. That said, for whatever reason Suunto pushes a bunch of pre-configured activity screens on us for a gazillion sports but limits our ability to create our own. Yes you can create custom activities and add most stuff, but you’ve got limited screens available and you can’t add things like graphs which are only available in the pre-configured screens.

GPS fix time is fast, so fast we really don’t think about it anymore (unlike years past when you’d be standing around in the rain waiting, waiting…). The Spartan Ultra uses the same bluetooth BLE standard as the Ambit3 and other non ANT+ brand so no issues there, though a low battery warning would be choice. And we’ve already covered the baffling foot pod situation in the previous section.

The Ambit charging clip is thankfully gone, replaced by a magnetic setup which is super easy to connect and hopefully not prone to the Ambit pin collapse syndrome. Though given the reduced specified battery life from the Ambit3, it would have been nice to offer a wearable charger like Garmin’s fēnix series.

Watch setup is a mix of on watch and Movescount configuration. Things can get bit confusing from time to time on where to go to configure what (eg. GLONASS is set on the watch via actvity options and applies to all activities, but touchscreen is set on Movescount and is activity specific), but that’s pretty typical for complex watches these days. PC based activity screen setup is a lot easier than the watch based approach, though it prevents any last minute changes in the field unless you’ve got a phone with internet access.

On that note Suunto appears to have upgraded their servers, as we don’t recall any signficant unplanned outages in 2017. Syncing to Movescount with the new SuuntoLink app on the PC is pretty quick and trouble free, though sometimes we need to reconnect the watch to the cable for it to kick off. Similarly the Movescount mobile app has proved trouble free in 2017.

While there is an ability to create interval training sessions, the functionality is pretty basic. Choose distance or time, choose a duration, choose a recovery, choose the number of repeats. No pyramids here then. The interval screen layout is also limited to a few preselected options, none of which match the field combinations we’d frequently use. So we’re back to button lapping and manually monitoring the set’s progress and intensity.

Sadly the Spartan series excludes the custom watch apps as used in the Ambit series. These turned out to be quite useful, mostly filling gaps in the Ambit feature set. With this gone we kiss goodbye to any version of race pacing, climb gradient, basic timer alerts (walk/run/eat/call your mum), limit alerts (HR/pace). And for a watch purported to be an all-singing-all-dancing, trail, ultra and outdoor adventure-time ready we are not sure why some Ambit3 features like storm and navigation alerts were dropped.

Long Term Verdict

So we try to avoid reading any reviews of the model before we start testing, kind of like a jury we want to avoid any prejudice. Though it was pretty hard to avoid the widespread knowledge that the Spartan had been released well before it was a mature, reliable, full featured product. While Garmin had already forged this beta (alpha?) route with the fēnix 2 & 3, the Ambits were solid from the get go and basically did what they said they’d do, the Spartan release clearly was premature.

We started testing the Spartan Ultra on FW 1.6.14 and consequently avoided some of the earlier issues regarding reliablity and broken features. The only reliability issues we’ve noted in over 5000km are three activities in the space of a week that self-aborted as the watch decided to do a mini-reset, and one teleport (like the GPS froze for 15min). In both cases it seems to have self resolved. Another issue to be aware of are that post-FW upgrade outings tend to be a bit wild (as is the case with most watches). We’d recommend recording a dummy outing by leaving the watch in place for 10 minutes or so before trying to record a move.

Like the Ambit3 we’ve had to re-pair the Movescount app periodically though this is thankfully straightforward these days. Other than this, the underbaked aspects of the Spartan Ultra relate to an incomplete feature set and poor trail performance positionally and distance wise. Both of these appear to be improving, as Suunto slowly adds features with each FW release, and FW 1.9.36 appears a little better accuracy wise.

That said, currently the watch still feels unfocused and incoherent. It’s not a full featured back country adventure watch, it’s trail performance is average at best, and it lacks features that would suit a full on racer. Sure it does the stuff most people would need most of the time, but you could get the same performance for under half the price. So where’s its edge?