bithacking - Tumblr blog

[bithacking]

Artifactory Online - Maven & SSO Authentication

Note to self (& others):  When using JFrog Artifactory (the SaaS aka “online” offering) and have configured SSO authentication with say, Google, you can’t use encrypted passwords as the mechanism for maven authentication.

Instead, you have to use your username of the SSO account, up to the domain, “john.smith” not “john.smith@my-domain.com”, and you have to use an API Key you generate from your account page as the password.

Those two become the <username> and <password> configurations in your maven’s ~/.m2/settings.xml.

There.  Maybe that saves you 30 minutes of futzing around.

[bithacking]

Auto-run nodejs tests

I like having tests run as I work on them, or as I work on the Code Under Test.  Here’s a quick 1-liner to get your mocha or jenkins-mocha based nodejs unit tests to run after you’ve made any changes to a given set of files.

Install fswatch (if you're on a mac, or inotifywait if you're on linux). brew install fswatch

Set the 1-liner to execute the test runner over the test you modified: fswatch -0 ./tests/bin | xargs -0 -n 1 -I {} ./node_modules/jenkins-mocha/bin/jenkins.js {}

Note: I'm using jenkins-mocha.

Now, when you modify any tests in ./tests/bin, fswatch will see those files change, pipe them into xargs, which will then run jenkins-mocha with only the file name that changed. This is useful if you want tests to execute every time you save a file, for instance.

[bithacking]

Custom Functions in Google Sheets

Here’s a quick use case:  you have a list of addresses in a google spreadsheet (sheets) and you want to know their distance (mileage) from one location to another.

You can write a custom google sheets function, GET_DISTANCE(), that will take in two addresses and return the distance from an origin to a destination in miles.  This function will reach out to the Google Maps Distance Matrix API with a server API Key you create (and tied to your google account).

If you are new to programming and scripting, the good news is that this is super easy. Lets walk through this.

Step 1: Create the Google Sheet

Create a new Google Sheet, and populate one of the columns with some addresses. Lets assume your origin address is the list of addresses you have in this column, and the destination address never changes (for example, your work office location).

Create a new page in the same google sheet document, and in any of the cells type in your destination (office location) address. While we are at it, lets name this cell using named ranges too. Right-click the destination address cell, and select "define named range". Give it a name, we can use this name to reference this cell later.

Step 2: Add a script

In the Google Sheet, click on Tools > Script Editor. Select "Custom Functions in Sheets". Delete everything on the page, and copy/paste this function in:

/** * @OnlyCurrentDoc Limits the script to only accessing the current spreadsheet. */ /** * Take two string addresses, and use the google matrix distance API to * fetch the mileage (distance) between the two. * * @param {String} origin An Address that resolves in Google Maps. The Origin. * @param {String} destination Another address that varies from the first and * resolves in Google Maps. The Destination. * @param {String} apiKey The API Key to the Google Matrix API, see: https://developers.google.com/maps/documentation/distancematrix/intro * @return {String} the mileage text distance between both addresses. */ function GET_DISTANCE(origin, destination, apiKey) { Logger.log("Fetching distance between " + origin + " and " + destination); var matrixHost = "https://maps.googleapis.com/maps/api/distancematrix/json?origins="; var encodedOrigin = encodeURIComponent(origin); var encodedDestination = encodeURIComponent(destination); var requestUrl = matrixHost + encodedOrigin + "&destinations=" + encodedDestination + "&key=" + apiKey + "&units=imperial"; var resp = UrlFetchApp.fetch(requestUrl); Logger.log(resp); var jsonData = JSON.parse(resp); return jsonData.rows[0].elements[0].distance.text; }

What this function does is take in 2 addresses an an API Key. It fetches from the distancematrix API the distance, and returns the text of the distance in imperial units "12 mi".

Step 3: Use the script

In your Google Sheets, go to a cell, and type in: =DISTANCE(ORIGIN,DEST,API_KEY) and you'll make a call out to the custom google sheets function that then returns the distance text and populates that cell with it.

The possibilities with Google Sheets and custom functions are endless. Between the vast treasure trove of Google APIs, and the fact you can extend Sheets with custom scripts, makes Sheets that much more powerful as an every-day tool.

[bithacking]

Publishing Artifacts to Bintray from Travis-CI

I came across a question recently in StackOverflow:

I would like to use Travis CI for my open-source project. The issue that Travis doesn't provide any ways to publish produced artifacts (though, they have this in their future plans).

What are workarounds to publish/upload artifacts somewhere? I'm allowed to execute any scripts on a CI machine.

Simple upload will work, but there is security issue: anyone will be able to upload something in the same way as all sources are public.

It was asked in 2012, and while a lot has changed since then, none of the answers recommended Bintray, which I think is a real shame, as it’s a repository system designed for this stuff.

I added my answer to the question, and thought I could expand more on the solution here.

Lets go over the basics:  you have an open source project hosted in github, and you want to tag releases, and have those releases build and make their way out to JCenter (bintray’s popular repository, and The Easy Way into Maven Central) for others to consume/use.

If you are using Java & Maven, this post is for you.  If you are using Java & <insert better tool here> then most of this post still applies, but you’ll want to do different deployment steps with your given tool.  Gradle is quite popular (and arguably better than Maven), and I’ve heard that process is easier, as there are better ways to do it with gradle.  Since my project was with maven, I’ll focus on that here.  Lets get started.

Step 1 - Create your accounts

You need accounts for Bintray and Travis-CI.  I’m going to assume you don’t have these, if you do, skip this and go to Step 2.

1.1 Travis-CI Setup

Head to https://travis-ci.org/ and create an account.  Link it to your github account, and give it access.  It should show a list of code repos in github that you control.  Select the repo you want, and enable travis-ci for that repo.

There’s more to it, so at this point, take a moment and read the getting started guide: http://docs.travis-ci.com/user/getting-started/

One last thing, before proceeding, read the section from travis-ci about encrypting secret variables (environment variables), we will do this in the next step so that when travis runs, maven has access to a username and password for the bintray publishing step:  http://docs.travis-ci.com/user/encryption-keys/

1.2 Bintray Account

You’ll need to create a Bintray account, and also create an API key that is associated with your account.

Go to https://bintray.com/ and create an account.  It helps to also register/link the account to your github account.  Once that’s done, head to your profile, and create an API Key that you’ll use later with maven (you reference an encrypted version of this as an environment variable in a custom settings.xml we create for maven to release with).

Next, create a maven repo for your user account (or bintray organization if you set one of those up).

Lastly, create a package in bintray that represents your project’s artifacts (published stuff goes in a “package”, each release is versioned and good to go).  https://bintray.com/howbintrayworks

During the package creation, you can also link it to JCenter, which will expose it to a larger public audience and let others more easily download and use it as a maven dependency.

Step 2 - Setup Maven for Bintray

Edit your pom.xml to include a distributionManagement section.  Looking at the <url>:  note that you need to fill in your bintray account/username, the repo name, and the project name.  The “maven” part stays the same, this is following the same examples that bintray provides:

<distributionManagement> <repository> <id>my-bintray-id</id> <url>https://api.bintray.com/maven/myUserName/myRepoName/my_awesome_project;publish=1</url> </repository> </distributionManagement>

Next, setup a custom maven settings.xml file that has your bintray username and bintray API key.  Note that you will first need to encrypt the BINTRAY_USER and BINTRAY_API_KEY variables using travis, (see this guide: http://docs.travis-ci.com/user/encryption-keys/).  Here’s an example settings.xml:

 <servers>    <server>      <id>my-bintray-id</id>      <username>${env.BINTRAY_USER}</username>      <password>${env.BINTRAY_API_KEY}</password>    </server>  </servers>

When travis runs, maven will have access to the decrypted environment variables.

Lastly, for maven, we need to add the release plugin to our pom.xml, so that we can do our releases (git tag, bump versions, make commits, etc):

 <build>    <plugins>      <plugin>       <groupId>org.apache.maven.plugins</groupId>        <artifactId>maven-release-plugin</artifactId>        <version>2.5.2</version>      </plugin> </plugins> </build>

Step 3 - Configure .travis.yml

Next, setup your .travis.yml yaml configuration file.  This file sets steps and configurations needed for Travis to run.  There are many guides for this, read these first before writing the .travis.yml file:

http://docs.travis-ci.com/user/languages/java/

http://docs.travis-ci.com/user/travis-lint/

http://docs.travis-ci.com/user/ci-environment/#Environment-variables

One important note, is that Travis provides an “after_success” section you can add that will execute after every successful build.  This is key for us, as we will add checks in this so that after every success we check for:

A git tag.

A specific JDK version (whatever version you want to publish from).  This lets you build under multiple JDKs but publish under one.

Anything else we desire.

Given those checks pass, we run “mvn deploy” to publish our built artifacts.

Here’s an example .travis.yml:

language: java jdk:  - oraclejdk7  - oraclejdk8 after_success:  - mvn clean cobertura:cobertura coveralls:report javadoc:jar  - test "${TRAVIS_PULL_REQUEST}" == "false" && test “${TRAVIS_JDK_VERSION}” == “oraclejdk7″ && test "${TRAVIS_TAG}" != "" && mvn deploy --settings travis/travis-settings.xml branches:  only:    - master      # Build tags that match this regex in addition to building the master branch.    - /^my_awesome_project-[0-9]+\.[0-9]+\.[0-9]+/ env:  global:    - secure: cfHTvABEszX79Dhj+u8/3EahMKKpAA2cqh7s3JACtVt5HMEXkkPbeAFlnywO+g4p2kVENcQGbZCiuz2FYBtN3KrIwFQabJE8FtpF57nswPRrmpRL+tWcYtipVC2Mnb4D7o6UR2PiC7g20/    - secure: cfHTvABEszX79Dhj+u8/3EahMKKpAA2cqh7s3JACtVt5HMEXkkPbeAFlnywO+g4p2kVENcQGbZCiuz2FYBtN3KrIwFQabJE8FtpF57nswPRrmpRL+tWcYtipVC2Mnb4D7o6UR2PiC7g20/

Notice that in the “after_success” part, I’m calling out to some other maven targets, and then testing for our conditions:  not a pull request, has a tag, and finally doing the mvn deploy (passing in my travis-settings.xml file from step 2 above).

Also notice that I white-list the build to only build on master branch, and any tag that matches the regex for “my_awesome_project”.

Step 4 - Make a Release

Finally, lets make a release.  For this, we use the maven release plugin from our local dev box.

Make sure you are caught up with #master branch (git fetch/pull).

Run mvn release:clean and mvn clean.

Now, run the maven release:

mvn release:prepare 

This will bump our version in the pom, removing -SNAPSHOT, tag it with the version we say to (use the default, so the regex in travis matches), push commits/tags, and then bump the version and add -SNAPSHOT, making a final commit so the latest version is in git.

Ultimately travis sees this tag, and our after_success tests pass, and a ‘mvn deploy’ occurs, which, with the settings.xml, sends the artifacts into Travis at our repo and package we set up.

After every release you run, do a local mvn release:clean, and git fetch/pull.

Final Notes:  The big end to end system

What do we get with this?  We get a system that looks like this:

Good luck, and let me know how successful this works out for you in comments.

[bithacking]

Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves.

Alan Kay, ACM Queue, http://queue.acm.org/detail.cfm?id=1039523