GPS Bumper Update Incoming (actually, truly this time)

Update – the initial Suunto Spartan Ultra trail review is up.

Yes, it’s been a while. The sabbatical in Italy put a dent in testing, sure. Then came a disc bulge which took any decent running load out of the picture. On the upside I took the time out to brush up on my geospatial-data-dorking skills in R and PostGIS and rebuilt the process entirely in R script. Yes, it’s nerd. The result is a complete re-analysis of collected data to give better insight into the capabilities of the GPS models tested.

In short, the updated methodology is leaning more towards quantified GPS performance bench-marking rather than a subjective review. Plus the future collection and analysis workflow is super easy which equal less typing and more running. Or beer, or whatever.

We can make pretty pictures like this track point cloud that shows GPS positional accuracy over a large number of runs (hint – this is a recent model GPS).

SpartanU_cloud_2017

 

Having powered up the analysis we can now see the impact of tree-cover, bendy paths, running speed, sampling rate, satellite availability, and even how much the satellite data is actually filtered out from the recorded distance (ie. does the watch work like a glorified activity tracker). The results give some great insights on which watches work best in actual trail conditions. With a couple of statistical tests we can formally identify which comes out as the better, or worse, performers. And the data can be modeled to show which factors effect watch accuracy.

As a teaser, the chart below shows the accuracy of all models across easy, mixed, and difficult conditions as recorded by the watch (ie. as you see it) in purple and as recorded by the raw GPX data (ie. buried underneath what you see) in yellow. The brown is the overlap of the two. The dashed lines are means, and grey is the true distance. We can see here the accuracy deteriorating between the conditions nicely. At the individual watch level there are huge difference in performance between these categories. Given we normally spend a lot of time running in the mixed to difficult conditions (when we are not testing) this analysis gives a great view on which watch will give us the good trail running numbers.

all_models_dist
The vertical purple line is average watch recorded accuracy, yellow is raw GPX (satellite) accuracy – is hidden behind the purple one in the bottom two panels

Plus, we’ve sold a kidney and promised a first born child to get hold of a Suunto Spartan Ultra for testing. The first round of formal surveyed trail testing is done, though sadly the +20hr endurance run test is off the cards for now (back prognosis points to a Spring running recovery). At the same time we tested a Sony Xperia Z5c using the SportTracker app. The results are interesting… our advice would be hold off any purchase till you read the review.

We’ll be writing up the Sunnto Spartan Ultra and Sony Xperia Z5c and updating existing reviews over the coming weeks. And data collection is in progress for the TomTom Adventurer.

 

Advertisements

MEC Ultra/Trail Runing GPS Test Champion (June 2016)

Ok, after a year and a bit we’ve finally exhausted the MEC’s pool of GPS trail running watches. This has included Garmin’s FR310XT, FR910XT, fēnix 2, and fēnix 3, Suunto’s Ambit2, and Ambit3 Peak, and Polar’s V800. Having put all these through extensive and objective real world trail testing we learnt a few things.

How to test

The context of testing is critical, just because it performs in a suburban park doesn’t mean much in the trail conditions you are likely to encounter in NZ. And sample size matters, some issues just don’t present after a weekend’s worth of fun. You need to repeat the test again, and again. And again. All the time controlling for conditions. Finally you need to test against some actual known truth, ie. a properly surveyed course.

Who to believe

Watch marketing is obviously heavily pitched towards the feature list and the vast majority of review sites do little more than extend the marketing reach of the brands. ‘Good’ reviews close to release date may shift product, but are ultimately meaningless if they don’t test what is important to you. And explaining features is not really testing, it’s like counting the tyres and calling it a race car. So ask yourself, honestly, is your review site of choice really testing or just teasing with a bunch of nice pics and gadget jargon.

We get it, it’s difficult and time consuming undertaking quantitative real world reviews with so many factors in play. But we care a lot about accuracy. We also are rather keen on reliability and the practical implementation of features (aka functionality). And true battery life in real trail running situations, we really do absolutely need to know if a watch can last more than 12hrs. Bottom line for us, performance equates to authenticity at this price point.

And you know what, it turns out you can’t take accuracy, performance, or reliability for granted. Even models from the same brand can fare very differently. We are reviewing these GPS watches simply because we want to see great trail watches from a variety of brands. That’s it. No sales agenda, no brand preference.

And the envelope please…

So our pick of trail GPS watches for NZ conditions? If accuracy, reliability, and endurance are your thing Suunto’s Ambit3 Peak is well clear of the rest. It tops all the accuracy tests we measure, and won the battery endurance test to boot. It’s not all roses though, the screen is a bit limited, there’s no vibrate alert, buttons are rather non-responsive, course navigation is basic (though still beats the competition with waypoint alerts on the breadcrumb course), and the battery savings modes aren’t very ultra friendly, and that charging clip! Also the Android mobile app is still not up to standard with daily re-pairing required (July 7 update – Ambit3 FW version 2.2.16 with app version 1.3.1 has survived a couple of days without losing connection).

See here for Trail Running GPS Review Roundup (June 2016) with links to each full review

If you don’t want to splash out big bucks and still value accuracy and endurance if you can find a Suunto Ambit2 or Garmin FR910XT you’d be well served (note the 910XT not the 920XT – which we haven’t tested).

And if you absolutely must have all the gazzilion features of the fēnix 3, or simply wish to support Garmin for their excellent openness towards data, you’ll be ceding a fair amount of accuracy and endurance (without a battery I.V. hookup). That said the fēnix 3’s screen, vibrate, and positive feel buttons make the user experience better than the rest.

So what now?

So now we’ve finished with the pool of available watches now what? Sure there are some more recent models, Garmin’s FT920XT, the fēnix 3 HR, and FR735XT. Yet I can’t see anyone in MEC ditching their current watches as none of those really add anything to the trail, and all have reduced battery life. Suunto’s pending Spartan Ultra looks intriguing, though with the touch screen and lack of detailed specs we’ll be waiting to see how that pans out before leaping in (especially at that price point).

In the interim we are reviewing Sony’s waterproof Xperia Z5 Compact with the Sports Tracker app (another Finn) as the combo gives barometric altitude and manual lapping capabilities. At present it’s beating about half the GPS watches in terms of accuracy. Looking forward to seeing if we can outrun it in the battery endurance test.

Also our lead tester, Ron, is about to go on a 6 month sabbatical in Italy so lots of running but not so much testing.

2015: A Year in GPS Trail Watch Review

After putting a bunch of GPS watches through some robust trail testing and long term experience with them we can do a ‘best fit’ for various runners. The runners described are –

Adventure and ultra-plus: battery endurance, accuracy in difficult conditions, reliability, navigation and back-country functionality.

Fast and furious trail racer: trail accuracy, reliability, and racing functions.

Budget trail runner: trail accuracy, +12hr battery endurance, and anything else we can get

Smartypants trail runner: trail capable, +12hr battery endurance, smartphone features

Roadie with a bit of trail on the side: like the fast and furious, but with less trail features.

The watches in the 2015 test pool were: Garmin’s FR310XT, FR910XT, fēnix 2, and fēnix 3, Polar’s V800, Suunto’s Ambit2, and Ambit3 Peak.

Note: not all these unit’s have full write ups yet but they’ve all done the time. Write ups are on their way.

Adventure and ultra-plus

First pick: Suunto Ambit3 Peak, unrivaled in the category. More accurate with more endurance than anything else tested, combined with absolute reliability and best navigation functionality.

Runner up: Suunto Ambit2, like the newer Ambit3 Peak but with less endurance and a few less features.

Fast and furious trail racer

First pick: Garmin FR910XT, not quite the match of the Ambits in terms of accuracy (is prone to extended track shadowing), but has a more race friendly design and feature set (screen legibility, vibrate alerts, button placement/feel, virtual pacer).

Runner up: Suunto Ambit3 Peak, while a more capable unit in most areas, it just loses out on the specific demands of blurred vision racing.

Budget trail runner (if you can find them)

First pick: Garmin FR910XT, pretty obvious really. Is more accurate with more proven endurance than the fēnix’s. Can’t be used day to day, and is prone to a bit of off-track meandering from time to time.

Runner up: Garmin FR310XT, pretty much like the 910XT without the altimeter. Also is orange and a bit bulkier.

Note: If there were a market for 2nd hand Ambit2’s they’d be worth a look too.

Smartypants trail runner

First pick: Garmin fēnix 3, more features than you can count on both fivefingers. Great looking, user design is excellent, and pretty impressive smartphone integration for a trail beast. Is held back by poor real world battery endurance and middling accuracy in tree cover.

Second pick: Polar V800, solid smartphone notifications, nice screen and easy to use design. Doesn’t quite cut is a full featured trail watch though, pretty accurate outside tree cover but stated distance a bit variable in the trees.

Road with a bit of trail on the side

Too hard to pick.

Suunto’s Ambit3 Peak is super accurate, has some great road running metrics, screen a bit average and lacks vibrate alerts, advanced workouts/interval training dependent on mobile app which still a bit iffy on Android.

The Polar V800 is fantastic on the road and urban trail, save the bulky pods, some proprietary issues, and general lack of trail features. Also often slow to acquire GPS.

And the Garmin FR910XT is still a class act on the trail and road, though can’t be used as a day to day watch and acquisition time is a bit painful in comparison with modern caching GPS.