tonyeagleone-blog

[tonyeagleone-blog]

Automated Testing Without Writing Code

In this article, we will be reviewing alternatives for Selenium WebDriver.

The reason why some companies actually prefer to use enterprise solutions instead of Selenium is due to stability concerns and long-term development.

Actually, I was surprised to find out that most Fortune 500 companies use code free enterprise solutions for automating their tests. While Selenium WebDriver is preferred by companies which are strictly focused on Software Development (such as Google).

For this review, me and some friends of mine from uTest gathered to test out these different enterprise solutions, to see which are actually worth checking out.

Our goal was to create some automated tests for a complex web interface. We came up with a number of 6 test cases and the goal was to create stable and robust automated tests with each solution.

Without further ado, these are the solutions that we have tested:

1. Endtest

I’ll just start directly with the winner.

We managed to create the automated tests in less than 2 hours, we were also able to run those tests directly on their cloud infrastructure.

Their interface is pretty clear and easy to use, we didn’t even need to check out their Documentation section.

You don’t have to download or install anything, everything is done in their cloud, we just signed up and started working on the tests.

Everything went pretty smooth, we did get a few errors when running the first drafts from the tests, but that was mostly because we weren’t targeting the correct HTML elements.

Another thing we really liked was the possibility to run our tests across different operating systems, browsers and mobile devices.

Even with responsive design and ever-improving standards support, cross-browser issues are not a thing of the past. Since it’s neither possible nor feasible to manually test your site in the galaxy of popular browsers and OS’s in broad use today, luckily Endtest came to the rescue.

One thing we didn’t like was the lack of Video Tutorials for using this platform, we understand that it was launched in 2016, but that shouldn’t be an excuse.

Here’s a short video demo for Endtest that we found on YouTube:https://www.youtube.com/watch?v=R4byO4dqOHU

After reading some articles from tech blogs, we found out that Endtest was a pretty huge surprise for the testing world, stealing important customers from competitors. We can understand why.

Surprisingly, it’s actually quite cheap.

They offer a FREE Plan which offers unlimited tests runs and a PRO Plan ($49) which offers some advanced features (such as granting access to your automated tests to other team members).

We sent them a message via the Contact Us section, asking a pretty generic question and they replied in less than one hour. We also contacted them on Twitter to report a minor issue and their reply was pretty friendly and useful.

Toyota, Ford, Vodafone, Dell and Intel are some of their most important customers.

What made this solution a winner in our eyes was that it’s a complete stable and robust all-in-one platform for automated testing.

2. Tricentis Tosca

Second place goes to Triscentis Tosca.

While their solution is not actually very stable, it does have some interesting options and it might be an interesting alternative in the future, if they fix their issues.

Their website is absolutely confusing, full of buzz words and enterprise non-sense. It took us a whooping 5 minutes to find the Download link for this application, we had to dig through a lot of ‘Success’ and ‘You are the future’ corporate titles.

See what I mean?

But we didn’t take the usability of their website into consideration while doing this review, since their solution is a downloadable Windows desktop application.

After downloading and playing a bit with the their app, we started automating our test cases.

We were very excited about their Test Recorder, we were hoping that it would we would finish in 10 minutes. But we were a bit disappointed to see that it relies mostly on image recognition (Sikuli? Anyone?), which sometimes isn’t very useful at all.

After trying to create the tests with the recorder for 1 hour, we gave up and used their visual test editor.

The visual test editor is pretty heavy, it has tons of options, but most of them are useless if you can’t even get something basic done.

Sometimes, the tests would just randomly fail, their mouse-controlling application wasn’t able to find them. It was very frustrating.

We tried for over 6 hours to finish the tests, but they were never stable, so we just gave up.

We used the Free 14 days trial, the Paid version is a bit too expensive for our pockets (it’s not even listed on the site, but for us it would have cost us 2000 US Dollars per month).

And you’re not getting any cloud infrastructure for that money, you’re just getting some downloadable Windows apps.

We sent them an email to ask some questions, but we didn’t receive any answer.

We contacted their VP of Operations and Product Manager on Linkedin, we didn’t get any answers. We started thinking that it might be a ghost company.

It’s amazing what a great Sales team can do, they do some impressive customers.

3. Ghost Inspector

Third place goes to Ghost Inspector, a pretty cool solution developed by a certain Justin Klemm.

Similar to Endtest, this is a cloud solution, so you don’t have to download or install anything to get started, except for the test recorder.

It does have a nice Documentation and useful video tutorials. Their User Interface is a bit too heavy for our tastes, but it’s decent (looks like your average weekend developer bootstrap.js site, which is just fine with us).

We downloaded the test recorder in the hope that it might save us some precious time. The recorder is actually a Chrome extension, which is not a really cool way of doing things (what if I’m using Safari or Firefox?).

When we played back the recorded test, it failed instantly, it could only access the URL and it couldn’t find our first button. Pretty disappointing.

It seems that no one in the testing world actually has a stable test recorder.

Anyway, we continued to write the tests using their visual test editor, which seems a bit built for developers. Because you just have to guess that you’re locating the elements by CSS Selectors and you have to do stuff through their API.

Why would you build a code-free testing solution for developers? Beats me.

After 4 hours, we managed to finish almost all of the test cases, except for the last one, where the test would just crash when it wasn’t finding a Submit button.

They do have a cloud infrastructure on which you can run your tests, but it’s mostly PhantomJS and Headless Firefox running on an Ubuntu server. It’s ok for a quick smoke test, but I wouldn’t bet my cross-browser regressions on their cloud.

We did use the Free Trial Plan, we wrote them a message about a possible trial for their Paid Plans, but they never answered.

We thought it was a cute little project until we saw the prices. Those prices are just way too expensive for such a basic solution.

To be perfectly honest, I think their solution would be more usable if they aimed their code-free product towards testers, and not developers.

We did try other solutions as well, but they were a complete mess (Ranorex, Sahi Pro, Test Complete), they were absolutely useless.

In conclusion, Endtest wins by far.