As a follow up article to my previous one about migration of JS to latest ES with help of Copilot I also tried to generate unit tests with it so lets break down what Copilot (or ChatGPT 4) can offer you.
For which frameworks Copilot can generate tests for you?
It seems that although default test framework for Copilot is Jasmine he is fw agnostic and can generate unit tests in :
- Jest,
- Mocha,
- Jasmine,
- Cypress,
- Enzyme,
- React testing library,
- Chai,
- Protractor.
In this article we will use Jasmine.
Can he guide you though installing all needed dependencies?
Yes, he can if you ask him to. It was a nice experience.
Can Copilot generate accurate tests?
Well… that is a mixed bag / legendary consultant answer - it depends. We will evaluate one fictional function and you will see.
Let’s say we have function like this:
function toUpperWithSharpS(input) {
if (input != null) {
let inputSplit = input.split(/[\ß]/)
for (var i = 0, len = inputSplit.length; i < len; i++) {
inputSplit[i] = inputSplit[i].toUpperCase()
}
return inputSplit.join("ß")
} else {
throw new Error("input param not specified")
}
}
As you can see this functions does one thing a bit differently from what is to be expected. In German language if you uppercase ß letter you switch it to SS. More on the topic here https://german.stackexchange.com/questions/2544/what-is-the-appropriate-capitalization-of-%C3%9F.
For our special purpose here we violate this rule but we want it like this. So let’s see what Copilot has to say.
If we select whole function and ask can you generate unit tests for me?
this is what we will get :
describe("toUpperWithSharpS", () => {
it("should convert string to uppercase and keep sharp S", () => {
const input = "straße"
const output = toUpperWithSharpS(input)
expect(output).toBe("STRASSE")
})
it("should throw an error if input is not provided", () => {
expect(() => toUpperWithSharpS()).toThrow("input param not specified")
})
it("should throw an error if input is null", () => {
expect(() => toUpperWithSharpS(null)).toThrow("input param not specified")
})
})
That is for sure what according to the rules should happen but we don’t want that, we named our function toUpperWithSharp
to explicitly state what happens here and we don’t want it differently for a reason.
I suppose AI saw this in training data and in some way one can argue it is right. OK so let’s try different way of generating tests, lets see what Copilot will say when we ask via context menu (select your function to test, Copilot > Generate tests)
describe("toUpperWithSharpS", () => {
it("should return null when input is null", () => {
expect(toUpperWithSharpS(null)).toBeNull()
})
it("should throw an error when input is not specified", () => {
expect(() => toUpperWithSharpS()).toThrowError("input param not specified")
})
it("should convert lowercase ß to uppercase SS", () => {
expect(toUpperWithSharpS("weiß")).toBe("WEISS")
})
it("should convert uppercase ß to uppercase SS", () => {
expect(toUpperWithSharpS("WEIẞ")).toBe("WEISS")
})
it("should not convert other characters", () => {
expect(toUpperWithSharpS("abc123!@#")).toBe("ABC123!@#")
})
it("should handle multiple occurrences of ß", () => {
expect(toUpperWithSharpS("weißbier und weißwein")).toBe(
"WEISSBIER UND WEISSWEIN",
)
})
it("should handle input with no ß", () => {
expect(toUpperWithSharpS("hello world")).toBe("HELLO WORLD")
})
})
Why there is different output from the same VS Code plugin, hard to tell.
Two weeks ago I was researching this same function for our company needs and the behavior was different between chat windows and between context command, now it is not. The chat window generated for me tests I would like to see - it did not escaped ß to SS.
Is there one simple input that can give me whole suite of tests?
For now is doesn’t seem we can do that. It rather seems Copilot will give you few tests and generate more tests when you ask him to. He seemed to ignore my question to generate comprehensive
test suite. Maybe there is some simple input that will do that. Maybe it works for now and it will not in 2 days time. Hard to tell. Maybe some visitor can ping me with what I did wrong here.
Is it good Copilot is not looking at implementation while creating tests?
IMO it should be default but I did not found a way to overrule this and generate tests which would consider comments or name of the function or some kind of hint from code. Yes, TDD tells us to create tests without knowing implementation, create the shortest code to make it green and fix edge cases if necessary, make it fast, iterate. Problem is when you have something special in the function and you need it to be like that. Then you need to fix the generated unit tests. Still Copilot can generate valid tests and test edge cases for you.
Can ChatGPT 4 create different tests?
While I was only aiming for GitHub Copilot it seems ChatGPT 4 (again differently from last time I checked for my company needs) can deliver unit tests in a way I would like to see them here.
My prompt : can you generate for me most comprehensive test suite for this function?
The result :
// toUpperWithSharpSTest.js
const { toUpperWithSharpS } = require("./path-to-your-function-file") // update the path to the file where your function is
describe("toUpperWithSharpS function", () => {
// Standard functionality with normal input
it("converts standard strings to uppercase while preserving sharp s", () => {
expect(toUpperWithSharpS("helloßworld")).toBe("HELLOßWORLD")
expect(toUpperWithSharpS("hello")).toBe("HELLO")
})
// Edge cases: empty strings and strings with special characters
it("handles edge cases correctly", () => {
expect(toUpperWithSharpS("")).toBe("")
expect(toUpperWithSharpS("ß")).toBe("ß")
expect(toUpperWithSharpS("helloßworldßagain")).toBe("HELLOßWORLDßAGAIN")
expect(toUpperWithSharpS("123!@#ßabc")).toBe("123!@#ßABC")
})
// Handling null and undefined separately as they are different cases in JavaScript
it("throws error when input is null", () => {
expect(() => toUpperWithSharpS(null)).toThrowError(
new Error("input param not specified"),
)
})
it("throws error when input is undefined", () => {
expect(() => toUpperWithSharpS(undefined)).toThrowError(
new Error("input param not specified"),
)
})
// Non-string inputs to check robustness of input validation
it("handles non-string inputs", () => {
expect(() => toUpperWithSharpS(123)).toThrow() // it should ideally throw a type error, but based on function implementation, behavior might differ
expect(() => toUpperWithSharpS({})).toThrow() // expecting similar behavior for objects
expect(() => toUpperWithSharpS(true)).toThrow() // non-string primitive
})
// Check with strings that contain characters from different languages
it("handles strings with characters from different languages", () => {
// Assuming the behavior is to keep non-English characters as they are
expect(toUpperWithSharpS("안녕하세요ßworld")).toBe("안녕하세요ßWORLD")
expect(toUpperWithSharpS("Приветßworld")).toBe("ПРИВЕТßWORLD")
})
})
So answer is - yes it can! I would expect this to be other way around, Copilot seems to me like AI focused on creating/understanding code but we maybe need to wait for some update.
If you know any other workflow I could write here down please let me know via email.
Have a good time generating unit tests (and reason about them) with Copilot.