Java with Maven: Giving CI/CD a try

Please note that this is work in progress.  I am still working on it and refining it, as my understanding of it improves.

I have a set of public repositories on GitHub showcasing my work, (⬀) which is in java with maven. These projects are interdependent, so when you check out one of them, in order to compile and run it you need the binaries of some of the others. You could manually check out all of them and put them in an IDE project, but that's too much work. Solving this problem requires having Continuous Integration & Continuous Deployment (CI/CD) in place, so I decided to try my luck in setting one up using free services only.

The process involves three entities:

• A Source Repository.  (Where our source code is hosted.)
• I use GitHub for this.
• Possible alternatives:
• GitLab
• BitBucket
• A CI/CD provider. (Where the actual CI/CD takes place.)
• I decided to use CircleCI for this, but in retrospect it was a bad idea, because it does not support GitLab.
• Possible alternatives:
• GitLab - I want to use it as a source code repository, and I don't want to put all my eggs in one basket, so I don't want to use it for anything else.
• GitHub - I want to use it as a source code repository, and I don't want to put all my eggs in one basket, so I don't want to use it for anything else.
• BitBucket - it is by Atlassian. Need I say more.
• Appveyor - gives various errors like "There was an error while trying to complete the current operation. Please contact AppVeyor support." -- Lots of open source projects are using it though, so it might be worth a second try.
• Travis CI - only works with github.
• JFrog - overwhelmingly fancy front page followed by a not particularly fancy user experience once you get past the front page. Once I have registered, there is no way for me to log back in. 
• semaphoreci.com - only works with github.
• buddy.works - after you have given them your e-mail address, they tell you that it is free but they require a valid payment method.
• atlassian.com/software/bamboo - it is by Atlassian, need I say more.
• drone.io - not only it works with nothing other than github, they assume that I am using github, which is very annoying.
• octopus.com - registration fails with "Please use your work email address."
• buildkite.com - might work; not particularly user friendly.
• codefresh.io - might work; they unnecessarily complicate things with mandatory docker images.
• An Artifact Repository. (Where the binaries are stored.)
• I found a place called Repsy for this; Repsy is minimalistic, unrefined, and they even have bad English on their web site, but it will do for now.
• Possible alternatives:
• GitHub Packages
• GitLab Package Registry
• JFrog Artifactory

We begin with a situation where we already have the Source Repository (GitHub) and we want to set-up the CI/CD Provider (CircleCI) and the Artifact Repository (Repsy).

Setting up the Artifact Repository

• Go to Repsy (⬀) and create a free account. The username in this case will be `mikenakis`.
• Create a public maven repository in Repsy. The repository name in this case is `mikenakis-public`.

Testing the Artifact Repository

Next, we need to make certain changes to our local setup so as to be able to deploy artifacts from within our local Development Environment directly to Repsy, to make sure this works before we even try CI/CD.

• In the maven folder (which is under `~/.m2` in Linux, `%USERPROFILE%\.m2` in Windows) create a `settings.xml` with the following content (or if one already exists, make sure it includes the following content)

<settings>
  <servers>
  <server>
<id>repsy-mikenakis-public</id>
  <username>mikenakis</username> <!-- Repsy username -->
<password>********</password> <!-- Repsy password -->
</server>
</servers>
</settings>

• Note:
• The username and password for deploying artifacts into Repsy repositories is the same as the username and password of the Repsy account.
• The password does not have to be included here as plaintext; it can be encrypted. For instructions on how to do that, see Apache/ Maven/ Password Encryption (⬀).
• Add the following to each and every `pom.xml` file that is going to take part in CI/CD:

<repositories>

  <repository>

    <id>repsy-mikenakis-public</id>

    <url>https://repo.repsy.io/mvn/mikenakis/mikenakis-public</url>

  </repository>

</repositories>

<distributionManagement>

  <repository>

    <id>repsy-mikenakis-public</id>

    <url>https://repo.repsy.io/mvn/mikenakis/mikenakis-public</url>

  </repository>

</distributionManagement>

That's it, we should now be able to execute `mvn deploy` in our local environment and deploy to Repsy.

Note: in reality I had many problems getting `mvn deploy` to work, because I have not really been using maven, I have only been using `pom.xml` files to describe my projects and then letting IntelliJ IDEA handle everything else, and it turns out that IntelliJ IDEA is a lot smarter and a lot more forgiving than maven is.  My `pom.xml` files needed a lot of re-working to get them to actually work with maven. But that's a different story.

Setting up the CI/CD Provider

• Go to CircleCI (⬀) and create a free account.
• Go to "Organization Settings" -> "Contexts" and create a new context; let's call it `my-context`.
• Add an environment variable to the context with name `REPSY_PASS` and the value being the password of the Repsy repository.  
• This way, our scripts will later have access to our Repsy password without us having to include it in any publicly visible source code. CircleCI promises to take extra measures to make sure that environment variables specified via contexts are never exposed to prying eyes.
• Create a CircleCI project which will correspond to one of our source repositories.  The source repository in this case will be `bytecode-dump`, so the CircleCI project will also be named `bytecode-dump`.
• As soon as we have created our project, CircleCI slaps us with their configuration editor, which they must think is very cool, and prevents us from going any further. We are not going to bother with it, because it is quite unhelpful, so:
• Click the "Use Existing Config" button.
• A dialog will pop up saying "Have you added a config.yml file?" and offering a couple of vague options that both seem to be dead ends. You might think you are stuck, but fear not and proceed to the next step.
• Select the "Start building" option.
• CircleCI will complain that there is no configuration, but that's okay, we will take care of it later. At least we can now access the project settings.
• Go to "Project Settings" -> "SSH Keys" -> "Checkout SSH Keys". 
• This is not about checking out any SSH keys, this is about specifying keys for CircleCI to be able to checkout.
• Click "Add Key". 
• You might think that you then need to do something with this key, but actually you don't. If you allow a few seconds to pass for the systems to do their magic, and then go to your project on GitHub -> "Security" -> "Settings" -> "Deploy Keys", you will see that a key will be there, (which though looks entirely different from what CircleCI shows,) and GitHub will say that this key has been added by CircleCI with authorization from you. Whatever. Magic.

Completing the configuration

• Back in our local development environment, in the root directory of our source repository (same level as the `pom.xml` or parent `pom.xml`) create a folder called `.circleci`.
• In the `.circleci` folder add a file called `mvn-settings.xml` with the following content:

<settings>

  <servers>

    <server>

      <id>repsy-mikenakis-public</id>

      <username>mikenakis</username>

      <password>${REPSY_PASS}</password>

    </server>

  </servers>

</settings>

• In the `.circleci` folder also add a file called `config.yml` with the following content:

version: 2.1

workflows:

  my-workflow:

    jobs:

      - build:

          context: my-context

jobs:

  build:

    docker:

      - image: cimg/openjdk:15.0.1

    steps:

      - checkout

      - run: mvn -s .circleci/mvn-settings.xml clean

      - run: mvn -s .circleci/mvn-settings.xml install

      - run: mvn -s .circleci/mvn-settings.xml deploy

• Note that we are supposed to add some additional notation to achieve caching of our maven dependencies, so that they are not all fetched from scratch each time the CI/CD build pipeline runs, but: 
• I have not found a definitive description of how to do this; 
• A magical incantation that I found somewhere initially seemed to work, but then later it caused problems, so I ditched it, and 
• CircleCI should really be automating this in a way which is totally transparent to us. So, for now caching will need to wait.
• If OpenJDK 15 does not suit you, then you will need to find another docker image, CircleCI has many available. I later switched to JDK 16 with no problems.
• There are 3 separate maven goals so that the `clean` goal can be easily commented out, and so that if the `install` goal fails then the `deploy` goal will not be executed at all. (Otherwise, rumor has it that you might get partial deployments.)

Firing it up

• Commit and push to GitHub.

Once these files have been committed to GitHub, CircleCI will take notice, and it will run our CI/CD pipeline. Do not expect it to run successfully on the first try; there will be errors, and there will be trouble. But once you get it to work successfully, it will be worth the trouble.