Meet ‘Codeball’ – A Deep Learning-based Automated Code Reviewer That Will Help Maintainers Review Github Pull Requests

Data in all forms, whether photos, music, or code, is being created on a large scale. Many GitHub code submissions require reviewers to read over the code and recommend modifications. Reviewing code requires a significant amount of time and effort on the developer’s part. Codeball fills this void. Codeball is a code review AI that approves Pull Requests that a human would have authorized. The AI detects and accepts safe contributions, allowing reviewers to focus their efforts on the difficult ones. Allowing shorter wait times saves a significant amount of money during the evaluation process.

Metadata from over a million Pull Requests and thousands of different repositories were used to train Codeball. Codeball extracts features for each PR using our proprietary derivation technique, constructing the bigger context in which the PR was filed. For example, how frequently and by whom were impacted files modified, the semantics of the diffs, and, of course, whether the Pull Request was accepted and merged without objections or further comments.

As a prediction model, Codeball uses a deep learning model that has been trained on over 1M public and private contributions from different organizations. In doing so it employs a Multi-layer Perceptron classifier neural network. In its input layer, the model takes hundreds of inputs, has two hidden layers, and a single output assesses the chance of a Pull Request being granted. Each Pull Request has hundreds of indications (the input layer). They are broadly classified into three basic, derived, and categorical types.

1. Basic — Metadata that is directly related to the Pull Request. For example, several lines may be added or removed.
2. Derived — Data that is related to the Pull request in some way. Some examples are the number of days since the last update to the file, file ownership ratios, file volatility score, and author-file score.
3. Categorical variables — Classifications derived from the Pull Request, such as file type categories.

The performance of Codeball’s model is always tested using code contributions that it has never been trained on previously. Precision — true positives / (true positives + false positives) — is the fundamental criterion optimized for. Codeball has an accuracy of more than 0.99.

The review consists primarily of pattern matching. According to the Ycombinator forum posts, in Codeball, the false-positive rate is less than 1%. Codeball has a 99 percent accuracy and a 25 percent recall. Codeball does not accept all potentially “approvable” PRs, and when it does, it is pretty sure that it will be accepted. Based on the working practices and the status of the repository, these values may be more significant or lower (ideally higher). The recall is a better way to account for this. Codeball has a recall of 48%, which means that only 48% of actually authorized PRs were expected to be approvable. To be cautious, Codeball mistakenly labeled 52 percent of the approvable PRs as un-approvable.

To get working with Codeball, one needs to add a file in the Github repo:

Create a new file called .github/workflows/codeball.yml with the following content:

That is all there is; Codeball will now watch all pull requests and approve the excellent ones.

As with other developing AI technology, some individuals welcome the shift, while others worry it will ruin the reviewing culture and may bring dangerous code into the codebase. It remains to be seen how code reviewers evolve because they can undoubtedly assist in reducing short-term expenses and review time.

This Article is written as a summay by Marktechpost Staff based on the Codeball page, reddit forum post and ycombinator forum post. All Credit For This Research Goes To The Researchers of This Project. Checkout the Github action and FAQs.

Please Don't Forget To Join Our ML Subreddit