Testing GitHub Copilot: How good is it?

Testing GitHub Copilot: How good is it?

In this article, I will share you with all the steps that I followed to join the GitHub's Copilot Technical Preview and I will share with you my humble opinion about this tool.

What is GitHub Copilot

GitHub Copilot provides completions for a dozens of languages and a wide variety of frameworks, but it works especially well for:

  • Python
  • JavaScript
  • TypeScript
  • Ruby
  • Go

In short words, in very simple human language, GitHub Copilot is a very advanced code autocompletion tool that will suggest you code to solve your problem or complete your task based on the code you already wrote in the file, the programming language that you are using and the naming conventions.

1. Joining the GitHub Copilot Technical Preview

I decided to apply as a member of the technical preview through the official website of Copilot. Access is limited to a small group of testers during the technical preview of GitHub Copilot. Signing up doesn't guarantee that you will get access, but it's worth a try. You will be asked how often do you use Visual Studio Code, either daily, weekly or monthly. Some devs wonder if open source contributions may affect the eligibility but there's no proof of it as some new accounts without contributions have obtained access.

In my case, it took about 2 weeks to receive the email that notifies that now I'm part of the Github Copilot Technical Preview:

GitHub copilot email

2. Installing GitHub Copilot

Initially, it seems that GitHub copilot is available only through an extension for Visual Studio Code, which is available at the marketplace here:

GitHub Copilot Extension VSCode

Of course, if you didn't sign-up for the Technical Preview, you will see an error when trying to use the extension in your editor. After installing the extension, you need to explore the Getting Started guide of GitHub Copilot in the official repository. As of now, GitHub copilot is a technical preview, so if you didn't get the invitation, you won't be able to access the documentation nor the repository itself.

If you got the invitation and you are logged in with your GitHub account, you will have access to the repository! You need to accept the invitation first, but be sure to enable the two-factor authentication of your account, otherwise you won't have access yet:

GitHub Copilot Invitation

After accepting the invitation, you will be notified that you now have view access to the github/copilot-preview repository. In this repository, you can find documentation, walkthroughs, examples, and the latest resources you need to use GitHub Copilot.

3. Sign in with GitHub in Visual Studio Code

In order to use GitHub Copilot, you will need now to Sign in with GitHub in order to use the extension. You can login through a simple link that will be opened in the left bottom side of the editor clicking on the user icon:

Signin with Github VSCode

A new window will be launched and you will have to authorize Visual Studio Code to access GitHub:

Authorize VS Code

After authorizing, be sure to restart Visual Studio Code as at the beginning, the code suggestion didn't appear until I closed VS Code and started  it again.

4. Testing GitHub Copilot

It's time to test this magical tool by myself! The first suggestion to test the tool as mentioned in the GitHub Copilot docs is to create a new JavaScript file (or use an existing one) and type the name of a function so basic as calculateDiffBetweenDates:

GitHub Copilot Suggestion

As you can see, GitHub copilot automatically suggests the code for this function and you can insert it using Tab. If you don't agree with the provided solution you may select another one if it's available using the Copilot dialog or switch the suggestion:

  • On macOS, press Option+] (or Option+[).
  • On Windows or Linux, press Alt+] (or Alt+[).

If you don't like any of the suggestions you can simply press Esc. Copilot allows you to create code from scratch using plain english in the comments of your code, for example in JavaScript you may simply generate the code of a function without a specific name and does exactly what you need it to do e.g find the png images in x directory and paste it to the y directory:

GitHub Copilot Suggestion

Just, Wow. The function instantiates an array of files reading synchronously a directory specified by the input variable, then it iterates over every file item, filtering them by the given extension, in this case png images and copies it from the input to the output directory. And all of this just by a simple self-formulated comment that I wrote (Wow once again).

In this case, the code its amazing and does exactly what I need it to do, however it uses a variable that hasn't been initialized yet (fs) as it is the filesystem module of Node.js, however you may do it as well with Copilot just typing the name of the variable and it will automatically suggest the code that imports the module:

Node.js Import Filesystem Module Copilot

Copilot is fantastic and these examples may be not enough to let you know how good this tool is, that's why I included as well in this article how to try to join the Technical Preview of Copilot. So I highly recommend you to join the Technical Preview and test by yourself how this amazing tool will help you to optimize your daily labor as a developer.

Another useful case that some developers have found is the possibility of automatically translate text in JavaScript objects or JSON files used for translation (for example i18n files of Vue.js based project):

Copilot Translation Files

Will GitHub Copilot replace developers?

This question has been on everyone lips lately. Hell no. Don't get me wrong, GitHub Copilot is an amazing tool that will be in the future a "must have tool" on every code editor of every developer, because it's just great. However, it's artificial intelligence what's behind it and therefore it isn't totally perfect (at least by now). There are a lot of caveats that will prevent this from happening:

  • Although it seems to create a custom piece of code that does exactly what you need, it isn't always complete. You need a developer to analyze the code and determine whether it works in the context or not, it's necessary to find what's missing and fix what's wrong. You won't just deploy the code that GitHub copilot will suggest to production without reviewing it.
  • GitHub's Copilot has been trained on gigantic pieces of code publicly available in their repositories. Many people suggest that Copilot’s generated code is absolutely derivative work, however for repositories whose License is Creative Commons Attribution 3.0, it requires attribution for derivative works. So, until now, the legal ground is still unknown as there is no direct legal precedent in the US that upholds publicly available training data as fair use.

Remember the old StackOverflow's Joke: you don't make $120K/Year for copying code from Stack Overflow, you get paid for knowing which piece of code from Stack Overflow does the job.

This could interest you

Become a more social person