GitHub Copilot vs. ChatGPT: How do they compare?

GitHub Copilot vs. ChatGPT: How they work

Although ChatGPT and GitHub Copilot have distinct user experiences and workflows, they share some similarities in terms of their underlying technology. Both systems use large language models to generate responses and suggestions.

While ChatGPT's LLM is trained on human language data, GitHub Copilot's LLM, called Codex, is fine-tuned on a massive data set of source code and natural language text, explained Nikita Povarov, data analytics and machine learning (ML) teams lead at JetBrains.

"This fine-tuning process allows Codex to understand the syntax and structure of code," Povarov said.

Here is the response from OpenAI's Codex to a request by the author.

In contrast, ChatGPT is a general-purpose conversational AI platform that uses natural language processing to respond to user input. It is better suited for broader, more complex tasks.

Agur Jõgi, CTO at CRM platform vendor Pipedrive, said the most common application development use case for ChatGPT is understanding and writing code based on the task description.

Here is an example of ChatGPT explaining code submitted by the author.

"Overall, Copilot is a great tool for quick, tactical tasks, and ChatGPT is better suited for broader tasks. Like any early technology, they both have areas of improvement and will get better and more advanced with time," Jõgi said.

Comparing the strengths and weaknesses of GitHub Copilot and ChatGPT

Here is a rundown of the respective strengths and weaknesses of each tool.

GitHub Copilot

Strengths. GitHub Copilot excels at generating code snippets and suggestions based on the context of the code being written, said Jorge Torres, co-founder and CEO at MindsDB, a platform designed to democratize ML.

Copilot can suggest lines of code, variables and function names relevant to the context of the code, and it can assist with granular snippets. GitHub Copilot also acts as a code completion agent that can complete tasks that the usual code completion tools built into the IDE cannot.

As such, it can save developers time and help them code more efficiently. In addition, GitHub Copilot can learn from the code that developers write, thereby improving its suggestions and accuracy over time.

Another code-writing strength of GitHub Copilot is its integration with the IDE, making it more efficient for real-time coding than ChatGPT and providing a better user experience overall for programmers, according to Jõgi. "Since it's embedded into the GitHub ecosystem, developers can use it without switching contexts or opening additional tools," he said.

Weaknesses. GitHub Copilot's weaknesses include generating incorrect or inefficient code suggestions, Torres said. It also might not be suitable for complex programming tasks that require extensive knowledge and expertise.

ChatGPT

Strengths. ChatGPT's strength is automating customer service interactions and providing virtual assistance for various tasks, Torres said. It can also generate text-based content, such as articles, stories and summaries, which makes it useful for content creation.

Povarov finds ChatGPT helpful for generating a wide range of code responses and explaining code concepts. It is also more suitable for nontechnical stakeholders and more flexible: Users communicate through a chatlike interface with a helper that can answer virtually any question. This flexibility lets users clarify and reformulate questions, leading to more refined answers and distinctions.

Jõgi also uses ChatGPT to generate function and unit tests to validate outcomes. This can streamline test-driven development practices.

Weaknesses. In terms of shortcomings, Torres finds that ChatGPT might struggle with complex or technical language, and not always generate accurate or relevant responses. It also might not be suitable for applications that require real-time interactions, such as gaming or trading.

Some researchers found that ChatGPT struggled to achieve the same level of measures for correctness, consistency, comprehensiveness and conciseness as answers written by humans on Stack Overflow. In addition, they observed that 52% of ChatGPT's answers contained inaccuracies, 62% were less concise than human answers and 78% of answers suffered from different degrees of inconsistency with human answers.

However, users still preferred ChatGPT responses approximately 39% of the time due to their comprehensiveness and more articulate language style. The danger, researchers wrote, is that users make occasional mistakes by preferring incorrect ChatGPT responses owing to better-articulated results and seemingly correct logic presented with positive assertions.

Similarities between Copilot and ChatGPT

Both tools take advantage of OpenAI's GPT LLMs to generate results in different ways. They also suggest different paths for improving all kinds of tasks besides just programming. For example, Microsoft has rolled out a range of Copilots for various Office applications and is also integrating ChatGPT capabilities into its Bing search engine.

"They both represent the future of generative AI experience design, where AI-powered assistants can help us accomplish tasks more efficiently and effectively," Torres said. As AI technology continues to improve, he said, we will see ever more sophisticated AI-powered assistants that can understand human language and context more accurately and generate more complex and sophisticated outputs.

Both tools have demonstrated ways that generative AI can boost developer productivity by automating routine tasks. Both kinds of capabilities could eventually be integrated directly into the development tools.

"In the future, we may see the integration of an AI assistant similar to ChatGPT directly into IDEs, enabling developers to communicate with it and request it to perform routine tasks, such as checking out code from repositories, running tests or building code," Povarov said.

If something goes wrong, such as a test failure or a build crash, the developer could ask the assistant to identify the error and suggest how to fix it. The AI assistant could do so, thereby reducing the time and effort required to maintain the codebase.

Preview of the future: Copilot Chat

GitHub recently rolled out Copilot Chat -- previously called Copilot X -- as part of a private beta. The new tool combines chat and terminal interfaces directly into the IDE. It promises to automate more aspects of the development experience. For example, it can detect code changes and automatically suggest descriptions, called pull requests, to accompany software updates.

The new offering promises context-aware conversations to explain code, find bugs and suggest fixes. It will also personalize answers that are linked to official documentation. In addition, it promises to write software tests and identify missing tests that might be required.

GitHub might add new services such as security, docs and testing down the road. The company said it wants to take the time to determine the best way to provide these new features to customers.

Coding assistant alternatives to GitHub Copilot and ChatGPT

GitHub Copilot and ChatGPT are no longer the only games in town. Some coding assistants, such as Tabnine, actually preceded the recent buzz of using LLMs to generate code. In addition, other coding-specific LLMs have been developed that promise an improved ability to securely fine-tune the models on existing or proprietary codebases and enterprise coding practices. In no particular order, here is a list of promising alternatives.

IBM Watsonx Code Assistant uses pre-trained models based on specific programming languages to improve trust and efficiency. It can help enhance transparency by providing visibility into the potential origin of generated code, translate code from one language to another or modernize legacy code. Watsonx Code Assistant for IBM Z, planned for late 2023, will help organizations refactor COBOL into Java.

JetBrains has been weaving AI coding assistant and chat capabilities across its portfolio of IDE tools as part of a technical preview. The features are powered by the JetBrains AI service that connects users to various LLMs. The assistant needs to be installed separately from the JetBrains marketplace. It includes a prompt library, AI chat, documentation generation, code explanation and commit message generation. Ask AI is a new feature in JetBrains' Datalore collaborative data science platform that is said to simplify coding in Python, R and SQL by supporting code generation and modification through natural language commands.

Google has announced several coding assistance tools. Google Bard now supports code analysis and development. Google Studio Bot provides a conversational experience like ChatGPT for Android developers. Google Colab, a data science development platform, plans to use Google's Codey LLMs optimized for code to generate larger blocks of code and write whole functions from prompts.

CodeGPT is an open source coding assistant extension that runs within the Microsoft Visual Studio IDE. It can connect to a wide range of LLMs optimized for code on the back end from OpenAI, Cohere, AI21 Labs, Meta and Anthropic. It can explain, improve and create new code directly within the IDE. A premium service starts at $9.99 per user, per month, and supports context management, AI agent creation, GitHub repository synchronization and other features.

Cursor from Anysphere is a newer AI-powered code editor built around generative AI capabilities that combines elements of an AI copilot and chat interface to speed development workflows. It includes a tool for generating code with a particular dependency, and it can answer questions about a codebase, analyze third-party libraries and automatically debug code. There is a free tier for basic access; there are also plans priced at $20 per month and $40 per user, per month, based on access, features and control.

Coding assistant adoption: Proceed with caution

Developers need to proceed cautiously in adopting both coding assistants and chatbots, said Kevin Smith, technical director at systems integration consultancy Dootrix.

The first big challenge relates to data security. "If you're typing sensitive data into ChatGPT, you may be in violation of the law or in breach of nondisclosure," he said. It's also important to be clear regarding the terms under which you are using the tools and to invest in alternative plans or enterprise agreements where necessary.

The new tools also increase the need for accountability. These tools can help developers be more productive, but they shouldn't replace experts. "You should treat ChatGPT like you'd treat a junior developer; you still need to check its work, the results may still need some significant reworking, and in some cases, it might just be wrong," Smith said.

Future of coding assistants

Smith expects coding assistant capabilities to become more pervasive, much like how every word processor ended up with spelling and grammar checking. Every development tool will end up making use of AI to improve developer throughput.

The tools will likely become more advanced as well through innovations in developer experience and coding LLMs. Smith also expects future gains from more specific models trained on specific data sets and problem domains.

In the long run, coding assistants and chat interfaces could also converge. "Just as we all gravitated toward the search engine and started seeing little search boxes in all our apps, maybe the chat interface will end up dominating and be the primary way we interact with AI," Smith said.

Editor's note: This list was compiled based on extensive long-term research into the field of code completion tools and analysis of some of the leading players, as well as newer entrants into this field. It should be noted that some consideration was also given to a new crop of LLMs being developed to support code assistant and chat capabilities on the back end, but not the front-end developer experience directly. For brevity and clarity, we decided to focus this article only on those tools that currently implement code completion and code chat capabilities on the front end and that work in conjunction with one or more code completion LLMs on the back end.