simplythetest

[simplythetest]

The Hazy Divide Between Manual and Automated Testing

"Are you a manual tester or an automated tester?"

I've heard this phrase or something like it for most of my career. In many organizations, there's some kind of distinction between manual testing and automated testing. In some cases there may even be different roles for manual testing ("QA Analyst") and automated testing ("Software Engineer in Test"). Much ink has been spilled on whether this division is helpful or not, so I'll simply mention that this distinction exists in some form in a lot of places.

Something I've noticed is that this distinction is getting more blurry all the time, in ways that I didn't expect. Of course this has been true for a while but things like machine learning are highlighting the haziness of this divide.

Here's something I did today that really drove home this point: I opened up a white boarding app wanting to see if I could make a mind map. I couldn't find the actual controls to create one, so I typed in "Create mind map to test email" into an AI prompt, clicked a button and - poof - a reasonable mind map was created with branches like Security testing, Performance testing and so on.

The basic creation of a "manual" testing skill, creating a mind map to test an area of software, has been automated to some extent. Perhaps the manual is becoming the automated.

I've seen "automated" testing is being automated with things like low-code and no-code tools and code generation tools. While complete test code generation fully replacing SEIT work is I think a computer science research fever dream, these tools for test automation have come a long way. Now imagine an individual on a team who can operate both manual and automated testing via automation, which is not an unlikely scenario.

Also note that there's nothing to say that someone without a job title including "quality" or "test" could use the above tools to do their work.

The wide gap between manual and automated testing is closing. What happens next could be anyone's guess.

[simplythetest]

Testing Specialists Shouldn't Advocate for the Customer

A long time ago, when I started in software, I read about what software testers should do as part of their roles. One thing that came up was the notion of testers being advocates for the customer. In this model, testers would find issues with software and advocate that certain issues be addressed and remedied "on behalf of the customer". After all, a tester can find issues, and who else can defend what the customer wants and needs better than a testing expert?

It turns out that a bunch of different people could advocate for the customer, and should.

For example, a product manager or product owner may regularly get direct feedback from customers, through interviews or surveys for example, where actual end users of software tell them exactly what they do and don't want. Customer support engineers see firsthand problems customers run into and help address those issues, possibly feeding information back into product development teams. And let's not forget our dear friends in customer success and sales, as these people regularly talk to customers and potential customers about all kinds of things. These roles have the ability to advocate for the customer since they actually know what the customer might want.

Sometimes it feels correct for testers in development organizations to want to advocate for end users. In some cases, testers are the end users. A tester who works for a bank and uses an online portal for that bank as a customer might have some insights in actually using the software. This might be even more true for software such as test case management or test automation, if you find a tester working for a company creating such software. Often, this isn't actually the case: a tester may be an excellent software professional contributing to software they will never use. In fact, they might not even come into contact with anyone who does use it. I've worked on a couple of software products that I certainly will not use for one reason or another.

The reason I point this out is that testers who try to advocate for "the customer" might have good intentions but actually not be helpful. Guesses - even educated ones - might lead to addressing quality issues that don't really exist. This wastes time and effort for everyone. Arguing with folks in product over hypotheticals can be a real drain on resources and enthusiasm. It also can create or worsen the perception of "testing as bottleneck" that slows everything down, since testers may chase unnecessary issues.

Naturally I believe quality is everyone's responsibility, but good teams work together.

[simplythetest]

Programming is Fun Again

This past weekend, I had the distinct pleasure of attending PyBay, a Python conference in the Bay Area. PyBay is always good, with excellent speakers and an even better community of people who come together to geek out over Python.

One thing that I took away from this year's PyBay was the idea of having fun - and only fun - with programming. This is something I've been thinking about for the past year or two. I've been getting into making art by painting over the past two years or so, in part because painting does not make use of computers. I realized over the years that I associate computers generally with "professional" activities, and not "artistic" activities. This distinction has worked well for me (hello test automation) but I'm learning that computers can be used for fun things in addition to professional business things.

This past week at PyBay had two presentations that showed how you can use computers - with some Python, incidentally - for fun. The first talk was Playing Improv ...with Python! by the wonderful Pamela Fox. This talk effectively showed how to play improv games with either human players or learning model players. On top of being a nice introduction to chat-based LLMs, Pamela also showed that LLMs can be used for fun and silly things. It was a great talk.

Another talk that got me thinking of play was Scrolling Animated ASCII Art in Python by the enjoyable Al Sweigart. Al showed various example of "scroll art", which is ASCII-generated code printed to a console line-by-line. Partly animation, partly word art, scroll art is totally cool. Al also memtioned that new forms are art are always being created, and that scroll art is a straightforward approach to teaching programming on virtually any computing device, which ties in nicely with the low-fi artistic aspect.

In both cases, the goal of programming or using software was to have fun. In particular, scroll art is based on barebones programming constructs: only loops, conditionals, variables, and ASCII or Unicode characters are used to create scroll art programs. The goal is to create something aesthetic and enjoyable. Computers can do that.

The theme of making computers fun again does tie into the greater Python ecosystem and community. There is definitely a good community vibe when it comes to taking chances and trying out new, fun ideas. Python allows humans to be human, on some level.

As a test developer, I've often been on the other end of computers, working to make sure that programs function properly for mostly commercial customers. Like other programmers, I learn specific tools, languages, and frameworks in order to solve business problems. I enjoy this as a career and frankly it's a pretty good one. But that doesn't mean computers are always for work; they can be for fun, too.

[simplythetest]

JavaScript In Anger: Global Test Timeouts

For the past number of years, I've worked with a variety of NodeJS-based test frameworks. These frameworks include Mocha, Jasmine, and Playwright. Interestingly - to me at least - these frameworks tend to have patterns and constructs in common. While some of these patterns are good (using a config file by default, for example), there is one that I frankly hate: implementing a global test timeout, and that timeout is turned on by default.

Dear reader, I don't like this feature. Not in the slightest.

The idea is straightforward: there is some timeout value - let's say 30 seconds - where if a test run's total execution time exceeds this timeout, the tests either fail or raise an error with a message along the lines of "your test run has exceeded 30 seconds". In many cases, this timeout is by default around 30-60 seconds. Having such a feature configured this way is, flatly, bonkers.

Test frameworks come in all shapes and sizes, just like software projects. Some are lean and mean, with a small number of relatively fast tests. Others are large and slow, with test code that is far from optimal. If you are creating a framework for test automation, you should anticipate each of these kinds of projects might use your tooling. For some, it may take 30 seconds to set up and connect to the system under test. Your test will fail out before it even gets started.

Remembering the power of defaults in software, a default value can greatly influence how your tooling works. A failed test is a failed test, regardless of the reason. If the reason ends up being that someone on a different team years ago decided that 30 seconds is "too long" for a test run, then that can lead to some awkward conversations. Coding frameworks should help your team and work, not be a stumbling block. Having a test fail because it took longer than some unclear value seems like a violation of the design principle of least astonishment.

Of course there are cases where you can disable this timeout, but that just makes me wonder why it was enabled in the first place? For example, in Jasmine there is a DEFAULT_TIMEOUT_INTERVAL value that is the max time a spec can take before timing out and failing. But why not disable this by default? How long should an arbitrary spec take to run? It's a completely open question based on context. These are questions and discussions that could be avoided simply by not allowing this feature to exist in the first place.

I will say I can see some good use cases for a global test timeout. In mature continuous delivery pipelines, tests that exceed some threshold of runtime could be taken out of the pipeline and re-evaluated, but this would require other pieces in place. A global timeout could also be a rough signal of performance problems, but again, other checks and balances would likely need to be in place for this work.

Having this timeouts like this enabled by default are something that do not spark joy in me, and I think this is the same for many teams working with JS testing for the first time.

[simplythetest]

Enable, Execute, Expect

Much ink has been spilled (metaphorically) around the topic of writing good unit tests. One of the best patterns I've seen for thinking about how to write good unit tests is the Arrange-Act-Assert pattern, also known as Triple AAA tests. Here's a selection of articles illustrating this pattern:

The Automation Panda's Take on AAA tests

Jay Cruz and the Three A's of Unit Testing on Dev.to

Leaning unit testing from Microsoft

and so on.

The gist is that every unit test should have three parts: arrange your test with fixtures and elements to test, perform an act on these fixtures that you wish to test, and then assert the actual result of the action is the same as you expect.

This is a great pattern, except for one tiny issue: in some languages, there isn't really an "assert".

Increasingly, instead of the syntax

assert expectedCondition == actualCondition

we see something like

expect(expectedCondition).toBe(actualCondition)

This is true in JavaScript/TypeScript world, as well as in the Ruby world. In JS you have this pattern in libraries like Jest, while in Ruby the RSpec framework uses expect as well. This could lead to developers who never actually use an assert keyword in code. This leads the Triple AAA mnemonic to shambles.

Hence, here is the recipe Triple EEE tests:

Enable test fixtures as needed

Execute a step to be tested

Expect a result and verify this happens.

A nice acronym, exactly expected.

[simplythetest]

Ask Why But Don't Expect An Answer

Lately, I've been thinking about code, maintenance and some other bigger thoughts. Often, things will occur over the course of working with software that makes us go "But why?". Asking why something is the way it is a necessary skill for a software tester, but like other skills it should be learned and worked on over time.

A long time ago, JB Rainsberger wrote something about working with legacy code. He was working with a piece of old code that made him go "why would someone write code like this?" or similar. (I'm not able to find the exact post of his.). His advice however was to ask "why", but not expect any answer. After all, the programmer who wrote the code was no longer there to answer it, and the answer wouldn't help the refactoring effort anyway.

I've been reflecting on this, and it strikes me as great advice for testers, developers and a bunch of other people as well. Asking why something (like code) was done a particular way is a natural question for people to ask. It can help provide context, as well as satisfy a need to understand the problem better. However, sometimes the explanation doesn't exist or isn't helpful. "Time pressure" doesn't exactly euclidate the problem, nor does reaching out to someone long gone from the situation.

Asking why is always something we can do to help us solve a problem, but we shouldn't always expect an answer, let alone a good one.

[simplythetest]

A Short Digression on Test Data

Lately I've been thinking about test automation and test data. I've come to a rather philosophical opinion.

Test data is always the problem, and test data is never the problem.

Test data is always the problem: Reading posts on generating test data for unit tests but also how to wrangle test data for various environments shows a lot of conflicting thoughts. Some say to never use random test data, some say they love it. Most people agree having "realistic" testing data is helpful but often for security or practical reasons, user data can't be directly imported and used for testing internally. I think in part there's so much conflicting experience because test data is unique to each team working with. Trying to use testing data to test an online banking application is nothing like data needed to test a mobile video game, which are both nothing like needing test data for enterprise accounting software. Throw in the fact that automation and manual approaches also have different needs and you'll see even more confusion.

Test data is unique to every team's problem. Even similar domains or applications may have subtle differences. Teams need to work things out themselves eventually.

Of course there's more to testing than simply data management, which leads to the second part of this, that test data is never the problem.

One of the biggest challenges I've had as a test automation specialist is getting people to understand what test automation is and why it's valuable. Even when I can achieve this and get support, test writing and strategy are still tricky sometimes. Getting developers - and their product managers! - to test and to test well are often bigger problems than whether we have good test data or not. In the words of wonderful Gerald Weinberg, it's always a people problem.

All problems with testing are due to problems of test data, until they aren't.

[simplythetest]

The Possible and Impossible: Mathematical Thinking for Planning

Sometimes, even knowing what is possible can be big step in the right direction.

In the past I've written about impossibility theorems in mathematics. These are class of theorems that definitively conclude that, given a set of constraints and conditions, something is impossible. A straightforward example is the impossibility of tiling the plane with regular pentagons. Try finding some regular pentagon shapes and making them tile a floor; you will find you cannot do this.

On the flip side to this, there is another class of theorems in mathematics called existence theorems. These are theorems that show that, definitively, something exists given some assumptions. Existence theorems may not provide any information on how to figure out how to find that thing that exists, or even provide a clue to what a value is. They may not even guarantee there's only one of the thing that exists. The only information an existence theorem (in general) provides is that a thing exists. One example is the contraction mapping theorem; basically if a function is a contraction - ie loosely speaking, the function's input maps to a subset of the input, "contracting" the input to a smaller set - then iteratively applying the function will produce at least one "fixed point" or a single point the iteration converges to.

So what does existence and impossibility theorems have to do with, well, anything other than higher mathematics?

Human beings tend to like certainty. We like knowing certain things will happen or won't happen, and avoid situations with uncertainty. Impossibility theorems and existence theorems are ways to figure out that, yes, something is certain. Whether that thing is certainly impossible or certainly possible may help us reckon with what to do next. This certainty may provide clues to how to find a solution, or to stop looking for one, or even to reformulate our assumptions.

In any case, knowing what is possible can be a big help, at least on a psychological level.

[simplythetest]

So You Want To Hire an SEIT

As software companies grow so do their engineering teams. A role that comes along with growing teams is the software engineer in test (SEIT), a software developer who primarily works with test automation and related testing infrastructure. SEITs can be extremely valuable to engineering teams as they can enable good automation practices for shipping quality code faster, and they can be a great touchpoint for test engineers and operations engineers to work with development teams. They can also be a bit tricky to hire since SEITs don't have quite the same skills as application developers or test engineers.

Let's take a look at how your might approach hiring SEITs at different levels.

Junior/Intern

I've worked with several interns (in Ontario these people are called co-op students) and working with test automation is a great introduction to being a software engineer of any kind.

When to hire

Juniors are great on teams with established SEITs and test engineers. In some organizations there are entire teams dedicated to test automation efforts. If you need someone to bang out many test code or take partial ownership of a large test automation project, a junior hire could be a great choice.

Like in other software develoment roles, hiring a junior usually comes with an expectation that the junior candidate will receive some menitorship and early career training and experience.

What to look for

Candidates for a junior SEIT should show characteristics of a junior test engineer and/or application developer. On the testing side, an ideal candidate should show curiosity about software or systems, the ability to develop at least basic mental models of how a piece of software might work, and good communication skills, written or oral. On the development side, an ideal candidate should show at least a basic aptitude for programming from school or otherwise. They should be able to read, write and understand code in some programming language even a language that your organization doesn't use. Previous experience working in software development is nice but not a requirement. It can be safe to assume that junior candidates have worked with at most one test automation tool and likely none.

Key questions to ask

"How would you test a pen?": a classic test engineering question to access a candidate's reasoning skills, communication skills and question asking ability.

"Write a function to reverse a list in a given language": a basic but straightforward assessment of programming skills. The language can be set by the interviewer or chosen by the candidate.

"What is something you achieved that you are proud of?": a question to get some insight into the candidate's motivations and interests.

Intermediate

Teams getting started with test automation or parts of test automation could benefit from hiring experienced SEITs. Experienced SEITs can write test code but also set up needed infrastructure and coordinate with developers, operation engineers, and testers. Intermediate SEITs may also be able to start test automation efforts for new projects as needed.

When to hire

The main reason to look for an intermediate SEIT is when automation efforts exist but have grown past being managed part time by single individuals. For example, a team may have written some Selenium-based tests to test some scenarios, and at first a small number of tests can be handled by the team or specific members of the team without a test automation speciality. Once the number of tests grow to a certain size and/or complexity, that team may want to have a dedicated person looking after them. This could also be the case with automated performance tests, service-level tests, and so on. Or a team may seek to level up on test automation skills.

What to look for

Test automation is a speciality of software development, and as I've said before (if slightly reworded) SEITs are developers. Evaluating them as you would other intermediate developers isn't unreasonable. Look for one or more roles where they worked as SEITs, DevOps engineers, or other forms of automation. Note however that SEITs may have written many, many lines of code but aren't traditionally skilled in areas of programming. Most SEITs wouldn't be able to code low level algorithms like linked lists or binary trees and may have never really used map/reduce/fold approaches. Intermediate SEITs should be able to discuss tools they've used well, and strong intermediates can identify differences between tools and levels of testing. And they can definitely write tests; any intermediate SEIT should be able to write a few tests for at least one layer of an application, such as unit or component tests, API level tests, and/or end-to-end tests.

Key questions to ask

"Write three automated tests in a given language to test this class": This question assess a candidate's knowledge of test development code writng

"Here is an example login test. How would you improve it?": A possibly fun exercise that helps evaluate a candidate's knowledge and technical communication skills.

"Suppose a software bug is found using test code that you wrote. What would be your first reaction?": Teams handle bug reporting in various ways with varying levels of success, so this question can shed some light on a candidate's experiences.

"Describe a time that you and a team mate - could be an app developer, tester or SEIT - disagreed. How did you resolve your differences?": Classic interview question for verifying what a candidate's personality is like.

"What's your favourite continuous integration tool and why"

Senior

If you work for a organization that has experienced growth and is now "established" in some sense, you may find some senior SEITs or other roles similar to that. In medium- to large-sized companies - companies a typical person may have heard of - there are often whole testing teams and sometimes engineering teams dedicated strictly to internal infrastructure. If you consider automated tests as infrastructure, this means you'll need dedicated folks to oversee project health. This includes writing and maintaining test code, but also includes managing people and computational resources. Good SEITs know how to put things together from a testing standpoint, and how to organize accordingly.

When to hire

This is definitely a bit of a "I'll know it when I see it" situation. Teams can often get pretty far with hiring and retaining good intermediate SEITs or a combination of SEITs, testers and DevOps folks. In many larger organizations, test frameworks can grow to require their own dedicated team leads and managers. This is often where senior SEITs come in. Another scenario is where test automation efforts expect to ramp up, and having someone knowledagble in test automation can guide the process.

What to look for

Senior SEITs have worked with multiple forms of test automation at most or all layers of an application. They will likely be well versed in more than one area of test automation, and have built test frameworks from stratch and worked with existing test frameworks. Depending on the circumstance, they also may have some team lead or engineering management experience, and so may be able to work with more junior members on a team in addition to coding and configuring infrastructure. One of the most valuable aspects that senior SEITs can bring to the table is an informed opinion on test automation topics.

Key questions to ask

"Suppose you were asked to review a test suite of 1000 tests. How would you approach this?": This question gets at how a candidate approaches a relatively complex situation. The idea here is spur discussion more than provide a flat answer.

"What's your preferred test framework and why? What's your least preferred?": In light of hiring someone who brings experience and opinions, this question asks for some of those opinions directly. It can also be a good indicator of communication abilities and temperament.

"Describe a time that you and a team mate - could be an app developer, tester or SEIT - disagreed. How did you resolve your differences?": Again, a classic interview question that could be critical if the candidate in question has to manage people.

"Are you willing to mentor junior app developers and SEITs?"

"Explain the value of end-to-end testing to me if I were the CEO of the company": Test automation can seem like an esoteric speciality to those not in engineering organizations. Senior developers of all kinds (in my opinion) should be able to talk in business terms in addition to technical terms.

Happy testing!

[simplythetest]

Breaking Down Problems Like a Mathematician

This week I changed a toilet seat on my main floor toilet. This was a substantial achievement, and I was able to do this thanks to my extensive mathematical training.

If you've ever changed a toilet seat on a standard toilet (in North America, anyway), you'll know it is fairly straightforward: undo the screws or latches on the old seat, remove the seat from the porcelain, remove any screws or nuts, and attach the new seat. I've done this a few times in my life and it is not too difficult.

However, this toilet seat was not easy to replace.

It was a style installed by the previous owners of my house that did not have easy to remove latches or nuts. The hinges were chrome and only attached by thick screws that went through the porcelain and had plastic nuts that held the screws and seat in place. The nuts were fastened extra tight, and could would not budge. It took me multiple attempts with multiple wrenches, and I could still not remove even one nut.

The turning point was realizing the entire problem (remove the old seat and install the new one) depended on solving one small aspect of the problem (remove one nut from a bolt). If I could solve this one small problem, I could solve the entire larger problem.

In mathematics sometimes a large problem can be reduced to a smaller problem or to a set of smaller problems. For example, properties of square matrices are often related, and knowing one aspect of a square matrix - for example, that it has a determinant of 4 - means you know lots about the matrix in question. With a determinant of 4 (aka not zero), we know the matrix has full rank and that it is invertible. Many problems in dynamical systems theory come down to knowing the eigenvalues of a square matrix, so even knowing what the determinant is can help provide information about the entire system.

In my case, I knew I had to remove the plastic nut. Using a hammer and chisel I hacked off the nut, which loosened the bolt enough that I could shift and remove the second nut on the other side, and in turn, remove the old seat. I was able to reduce my larger problem into a smaller one.

This shift in thinking also had a psychological benefit. All of a sudden a difficult multi-step problem became a focused single-step one. I moved the problem forward.

Math is surprisingly helpful to learn as a way to learn problem solving and thinking, even in situations that look like you're going down the drain.

Friendly Announcement: If you liked this post or ones like it, sign up for my math newsletter

[simplythetest]

Thoughts on Targeting Quality 2023

It's been a heck of a few years for me, personally and professionally. One thing that I am grateful for is returning to Targeting Quality 2023, the annual conference held by KWSQA. This year the conference was held in Cambridge, Ontario. Here are my thoughts on it.

Overall

The testing community is still a great community. Attendees were engaged, and even as someone who's done primarily test development for his career I was still able to have great discussions with testers about all kinds of topics.

The Talks

The first talk of the day on Presentation day that I saw was Bas Dijkstra's How well do you wield your tools? - The importance of craftsmanship in test automation which was a part of the automation track. One of the key messages in this talk was that test automation can and should be treated like production software development, which is something I've been saying for a long time. It is good to hear that this message is still coming through strong.

Test automation is definitely a mainstream topic in software development, but there's still so much that can be done by application developers in terms of test automation. Concepts like property-based testing, contract testing and security testing are all highly relevant for test development.

Fuzz testing: still not a common topic outside of the security engineering world

Another talk that I enjoyed was The Automation Firehose by Thomas Haver of M&T Bank. This talk gave a great overview of how to think about growing and scaling test automation teams with a superhero theme.

Ben Oconis' talk on breaking down silos in organizations sparked some thoughts from me and folks online about the role of "test specialist" in software organizations. Ben's talk was really good at outlining some organizational aspects to quality, which aren't necessarily "testing" skills but important nonetheless.

On this note, software testing is still a great place to learn about good team building practices. Lean Coffee, for example, is an excellent format for getting engaged folks to talk about a few different subjects in an effective (and time efficient!) manner. There were two sessions of lean coffee lead by Lisa Crispin and Janet Gregory. I attended one of these sessions, and I can't recommend trying lean coffee enough.

My favourite talk of the conference was the keynote, simply entitled STAIRS! by Jenny Bramble and Jenna Charlton. Entertaining and insightful, Jenny and Jenna show great examples of cognitive biases and how they can effect how products get designed using examples of stair designs (some of which were hilarious bad). This keynote showed that people in the testing community are digging into concepts such as design thinking and how that impacts testing work and quality, and doing so ahead of other in the software development world.

The Conference

I have to say: this is one of the best run conferences I have attended. Targeting Quality was 2 days this year, one day of workshops (which I did not attend but heard good things about) and one day of presentations.

The venue was well-prepared and suitable for the purpose, right down to stacks of pens and notepads available all over the place.

Food and drink was great, including delicious Portugese-style custard tarts at breakfast (a local delicacy, in my opinion)

The talks and schedule were well-organized and everything ran pretty much on time throughout the day. This is an underappreciated aspect of any conference, large or small.

Overall, this conference offered a lot of benefits for attendees and speakers. The hall-way track was lively, talks were great quality and speakers were well taken care of. The KWSQA board and Tina Fletcher should receive a round of applause for a great job.

[simplythetest]

The Task Is Not The Tool

Really, the title of the post says everything. Using a specific tool is not the task to complete. It is the tool.

This comes up often in software development, where folks like to discuss and plan around a tool instead of a task, but an analogy I've been thinking about is dishwashers.

I have a dishwasher and, by far, it's my favourite kitchen appliance. Dirty dishes go in, I turn it on, and a little while later, clean dishes come out. It works, it works well, and it does what I need it to do. One day, I was thinking about the role of being a dishwasher in a restaurant, which is a traditional "entry-level" role in the hospitality industry. I wondered: Why don't restaurants simply use one or more great dishwasher appliances instead of a full-time employee dishwasher?

This thought (which wasn't too well-considered) lasted a few minutes before I realized the answer: there's more to keeping dishes clean than washing them. Dishwasher employees need to clean guest flatware and tableware for sure, but also pots and pans and such for the back-of-the-house kitchen as well. These items need to be cleaned, and allocated back to where they should be, and this should be done in a timely way. Cleaning a dish is only one aspect of this job. The task is to keep items in a kitchen clean and ready as needed, where an automatic dishwasher appliance is a tool to help accomplish this task.

The same goes for test automation approaches, for example. Choosing Selenium over Playwright for browser testing is making a decision about a tool, but deciding what kinds of tests or aspects of a software development process should be automated is (likely) the task.

Keep this in mind when thinking about tools and tasks.

[simplythetest]

The Hottest Club in Town: Cybersecurity

If you're looking for a good time, check out the hot new tech subject, Cybersecurity.

(I'm channeling my best Stefon here.)

Seriously, this topic has everything. Two talks in this area from this year definitely have me thinking that when it comes to cybersecurity, the truth can be stranger than fiction.

Earlier this year, I attended a talk at BSides Rochester entitled "TikTok Under Attack: Attacker Uses a Popular TikTok Challenge to Lure Users Into Installing Malicious Package". I was intrigued because I do enjoy TikTok, but was not prepared for the roller coaster ride of this talk.

The overall story goes like this: the speaker is a security researcher who was interested in starjacking, a potentially malicious process where a given GitHub repository is shown to have a much higher number of stars/stargazers than it actually does. This can be exploited by poor controls on package managers such as NPM or PyPI which don't necessarily validate a GitHub repo's metrics. Here's a good intro to the topic.

This researcher was finding ways to understand how starjacking works, and found that there were malicious PyPI packages that if installed, will enable starjacking on particular repos. Installing a PyPI module does require some Python knowledge - it's not quite as easy as clicking a link on a webpage - so the attackers had to find a way to get individuals to install this mysterious PyPI package.

This is where TikTok comes in.

The researcher found a challenge on TikTok called the Invisible Filter challenge. The idea was that TikTok created a filter that makes a person in a video "invisible" (you can only see a rough outline of a human while the shape blends into the background). The challenge was to create videos and have viewers guess who the invisible person is, what they're wearing, etc. A variation of this challenge was for the person in the video to be naked so to "appear naked" on TikTok without actually appearing, er, naked. The attackers exploited this by posting links to their malicious PyPI package saying this Python package could remove the invisible filter showing the person in the video. This caused a bunch of folks (referred to as "creeps" by the speaker) to download and install the malicious package, thus facilitating starjacking.

That talk was a wild ride. Where it started and where it ended up were two different places.

I thought for sure this was a one-off style of presentation (PyPI! Starjacking! TikTok! Nude people!).

But wait, there's more!

During this year's BlackHat conference, a PhD student gave a talk called "Houston, We Have a Problem: Analyzing the Security of Low Earth Orbit Satellites". Effectively, the researcher asked "How secure are in-orbit satellites?" and the answer is "not very".

While I didn't see this talk myself, I can imagine this having the same flow as the TikTok talk. For example, the researcher found that many security features of satellites are non-existent, likely based on the thinking that attackers on Earth couldn't access them in any way. That turns out to be false, and even cloud services such as AWS offer Ground Service to low-orbit satellites. You can even get started for free.

Again, this talk had everything: basic hacking techniques, cloud providers, satellites...in space!

When I think of cybersecurity, I typically don't think of talks that offer stranger than fiction style narrative, but it may be the best place to find some such tales.

[simplythetest]

Doing the Work vs Finding the Work

Many years ago, I took two courses in graduate-level mathematics as part of a summer school program at Dalhousie University in Halifax, Nova Scotia. It was an enlightening experience that very much enjoyed, learning some math on Canada's East coast (I highly recommend visiting Halifax in the summer).

In one my courses, we were given an assignment to prove a result from probability theory. The proof was relatively difficult but not impossible, and those in the class were not necessarily experienced in intermediate probability theory (myself included). The professor mentioned that there was a proof of this result published in at least one textbook, if not other papers and books. He also said he would accept a reference to this proof from a source, instead of proving this result directly ourselves. This was the first time I ever had a mathematics professor (or teacher of any kind!) tell me that I didn't have to prove a result myself if I could produce a valid reference to a proof.

So I went to the main library on campus, did some digging, and found a proof in a textbook. I took down the reference, handed in my assignment, and received full marks for that section. Mission accomplished.

Engineers, software or otherwise, may have heard the phrase "a few hours of research can save a few weeks of development" and my experience with this proof was my first real experience with this line of thinking.

Sometimes there's no reason to re-invent the wheel if you can find the work done somewhere else.

[simplythetest]

Don't Build a Pseudo-Number Generator

This is a part of a series of things developers should not build.

Often, developers need to write code that has some element of randomness. In game development actions may depend on a dice roll or coin flip. In scientific programming and simulation generating random fractional values within a range (between 0 and 1, for example) is a common task. From a design perspective, perhaps a developer needs to randomly select an image from a set of images to present in a front-end. In any case, a developer may think "I know, I'll write some code myself to generate some random values".

This is -- with some narrow exceptions -- not a good idea and a developer should not do this and instead use a pre-built pseudo-random number generator library. There are libraries for this task in most programming languages. Failing that, there are CLI utilities that can be called from the OS level.

Let's take a look at why this is the case.

The Theory is Hard

First, what's the difference between random and pseudo-random anyway? Isn't this just a technicality?

The short answer is: no, it is not just a technicality.

Randomness is a difficult topic for human beings to understand. Truly random events have zero relationship with one another, meaning that one event has zero impact or influence on the next one. If I roll a six-sided die and it turns up a four, then rolling again has no impact on the next roll the next time if the rolling event is truly random.

Now consider about a computer program or script. It needs to run from the beginning to the end. Code runs in a predictable and deterministic way. By definition running a program from beginning to end cannot have any randomness. Computers can't stop and roll a die.

Luckily, there's a workaround: create a function for generating numbers that appear random, statistically. Such a function returns the same output for a given input. However, each generated number is "random-enough" to simulate a random event such as dice roll. Mathematicians have worked on creating algorithms and functions that output random-seeming values called pseudo-random numbers that have good properties for being close enough to random. These include algorithms such as linear congruential generators and the widely implemented Mersenne Twister algorithm.

(As a side note, if true randomness is a hard requirement in an application such actual lotteries using dice rolling, there are approaches that use physical phenomena to generate true random values. See RANDOM.org for more information.)

Given the theoretical issues, the creation of pseudo-random number generator (PRNG) algorithms provide a path forward. Sadly, PRNGs do not end the problems developers face.

The Practice is Hard

Implementing a PRNG seems straightforward. Find an algorithm, pick some values for various aspects of the implementation, and execute away. Many PRNG algorithms are effectively difference equations that start with a given value and iterate through a provided number of times. Easy right?

Well, no.

The first issue that arises is computation time. Generating a single value from a PRNG is fast, but what about generating 100 values, or 1000? In practice, applications require a random value on every command or execution which could mean that many, many random values need to be generated. If each generation requires many, many iterations of a particular algorithm this could lead to non-trivial execution times. In game development, this could be a deal breaker, since performance is crucial. Often PRNGs need to be fast at runtime.

Computer hardware has gotten fast thanks to Moore's Law so even executing a large number of iterations within a function is pretty fast. However, speed isn't the only consideration for PRNGs. There's also the aspect of being "statistically random" which means choosing a "good" PRNG with good configuration.

An infamous example of choosing a "bad" PRNG algorithm was RANDU. RANDU was based on a linear congruence generator as mentioned above. While RANDU generated approximately random values, it had some obvious flaws. For example, RANDU only generated odd numbers. As well, after a while generated values would repeat, meaning there was a pattern that did not appear random at all. Worst of all, RANDU was taken an implemented in a variety of settings. Note that this was a vetted algorithm by smart folks.

While it is definitely possible to test a random-number generator for correctness and performance, PRNGs are sensitive to configuration and there are many pitfalls that a developer may encounter if they are not familiar with the problem space of numerical analysis and computational mathematics. Which in my experience is not a lot of developers.

However, smart folks have learned from their mistakes and now there are off-the-shelf PRNGs using vetted algorithms in most mainstream programming languages.

It's all good, right?

Don't Forget About Security

The last and possibly biggest issue with using PRNGs in code is security. If an attacker can determine the PRNG values being generated by a program, that attacker can exploit this knowledge. A helpful example is an online poker site. If each card that turns up is chosen using PRNG algorithm, then a player who knows what value the PRNG will output next can use that information to determine every single card dealt to every player. This knowledge can easily be exploited to manipulate each hand.

One of the reasons that RANDU was such a disaster was that it was not secure and it was widely used. Once an attacker know a piece of software is using RANDU they can exploit that information for other malicious purposes. Using a PRNG in a local toy piece of code may be completely fine but using an insecure PRNG in production could lead to major security holes.

Again this is not a scenario that many developers consider when deciding to use a bit of randomness in their code.

Mercifully, there are well-tested cryptographically secure PRNGs available in many programming languages.

In summation, choosing a PRNG built and maintained by experts is a better bet than rolling your own.

[simplythetest]

Programmers, Don’t Build These

In the spirit of “You can’t buy integration” here are things that a programmer should never build:

A Pseudo-Random Number Generator

A Cryptographic Library

A Numerical Integration Solver

A Fuzzer

Each of the reasons why are straightforward: building each of these from scratch is loaded with pitfalls that could cause problems that far, far outweigh true benefit.

[simplythetest]

Choosing Developer Events: The Christoper Plummer Approach

How you decide what events to go to?

This is a tricky question for most people, but even more so for people who go to events as part of their job. For example, developer relations advocates often present at conferences to tell people about their cool projects (like for example a AI-guided fuzzing tool). However, there are only so many hours in a day, and only so many events someone can attend in a year. How can you choose the events that work best for you?

One approach that I think works well is to take inspiration from one of the greatest actors of a generation: Christoper Plummer.

The late but great actor Christopher Plummer took every kind of acting role you could imagine: large scale blockbusters musicals to European dramas to Shakespearian plays (and even some lesser quality films). When asked how he chooses roles in an interview, Plummer said a role needs to have one of three characteristics: the role is interesting enough, pays well, or is in a great location.

This is how I think about events, and I think the Christoper Plummer rule works well. Stated for choosing developer events, you should consider going to an event if the event

is interesting,

pays well, or

is in a great location

Interesting Role (or Event)

Developer events and conferences come in shapes and sizes and cover all kinds of topics, from the most narrow programming construct to the broadest area of industry. Depending on your interested or priorities, there are likely a few "major" events or conferences that take place on a regular basis. For the Python world there's PyCon, for the infosec world there's DEFCon, and so on. These events draw a lot of people and event simply showing up without a plan may still be beneficial. These major conferences and events are definitely something to consider, even if you've never attended or aren't overly familiar with them.

Pays Well

Sometimes conferences will pay for speakers to attend, and in some cases this includes food, accommodation and travel expenses. Similarly, most conferences look for sponsorship from potential partners and vendors and in turn provide conferences passes. If you have the inclination to speak to people, from onstage or from a table, you may be able to attend a conference at very little out-of-pocket cost. Who doesn't love free stuff? Finding opportunities to go to conferences at little to no personal cost might help make them more attractive to you.

An Interesting Location

Have you ever wanted to visit Disney? Then going to a conference or event in Anaheim, California or Orlando, Florida might provide the perfect reason to visit. Many places draw visitors for business or pleasure based on local culture, commerce or climate. If an event happens to take place in somewhere interesting to you personally, it might tip the scales in your favour of attending. A great location also makes a less valuable event seem a bit less bad.

If all the world's a stage, then consider these three things when choosing where you'd like to be a part of the play.