CyberChef and GitHubCity Contributions

GithubCity

For the first bug in the GitHubCity repository, I decided to tackle a bug regarding an if else statement. The bug can be seen here. What the bug is about is the code used for the if else statement of a function is overly repetitive and needs a cleaner way of doing this. There was a person who previously did this change and the admin was asking for someone to fix the bug similiar to this person.

As you can see in the code sample above, this code is written in Python. Also the if else statement is checking the values of order, and based on that will perform sorting operations with different parameters. The solution to reduce the amount of code provided to less than 20 lines is to use a built in Python function called getattr(). What getattr() does is it takes in 2 arguments, an object and a string name, and it will check if this string is a key in the object since the object is a list of key-value pairs.

If the name is a key in the object, then it will access the value of that key, and return that value. So using this getattr() function, the if else statement above was reduced to the code sample below.

So as you can see this try catch is used to ensure any exceptions can be handled by the getattr() function since it returns an AttributeError if the string name does not exist as a key in the object. Steps by which I used to actually implement this bug fix was I first had to fork the repository, clone it locally, create my own branch, and commit the changes to this branch. I made my Pull Request for this change and it can be seen here. I also noticed some of my changes were refused by travis.ci and this is regarding the tests that the admin had beforehand. I will continue to explore this issue and communicate with the admin regarding whether I am allowed to change his tests, or if he would make a recommendation to change my solution.

CyberChef

Onto my next bug and that is in the CyberChef Repository. If you are wondering what CyberChef does, I discussed it in my previous blog post however here is a link if you are curious. Below is a demo of how the web app actually works.

So the bug I decided to tackle in this repository revolves around implementing a new functionality. So currently the CyberChef can handle many different input types and formatting them, but it cannot handle CSV parsing. This issue was raised by an admin and it can be seen here. So the first step I did was to get some clarity about the actual purpose of the operation and whether it needs to work along with an existing CSV parser or if its completely new. So I decided to reach out to in the issue as a comment.

As you can see his answer wasn’t exactly what I wanted but based on my code research later on, I realized it was standalone and not going to be dependent on any other operation. The next step was to find an npm module to handle CSV parsing. The reason why I had to use npm is because the entire project is written in Node.js. After a quick search on Google, I found the package that I wanted to use. The npm page can be seen here.

Before I added the functionality to the repository, I decided to test it on one of my Node.js projects. This way I can easily run and debug the functions in a smaller repository without requiring the entire CyberChef repository to be frequently built. I decided to use my API application that was built for the 0.1 release. After reading the documentation for the functions in this csv-string library, I identified 3 functions that I was going to use. These 3 functions are parse, stringify, and detect.

First on the list, parse is a function used to parse an input string into an array format. It takes in 2 other optional parameters which are the seperators and the quotes. Seperators are what is used to seperate each column in a csv, however the default value is a comma as expected. The quotes is the ASCII character which is used to enclose string or other data inside of one column. Default value is double quotes. So I decided to pass in the input into this parse function to parse it to an array. The admin asked in the issue for either JSON or array of arrays so I decided to do array of arrays.

Stringify is used to convert an input string to a CSV string. I decided to add this functionality just to support converting to AND from a CSV string. It takes in an input, which can also be an array of arrays, and a seperator which has a default value of comma.

Detect is the last function I used to help automatically detect the seperator used in the input string. I decided to use this in case a user passes in a seperator such as a semicolon into the parse function, and the actual seperator detected in the input is a comma. This way the parsing will not break since it will then use the comma that was detected as the seperator instead of the semicolon.

After testing my function calls and making sure they work, I decided to fork, clone, create a new branch, and add my changes to my fork of the CyberChef repository. I also had to add some tests which will be shown a little later.

Now the most interesting part of this entire contributing process to CyberChef without a doubt is the files that need to be updated in order to implement a new operation. The documentation for how to do this can be seen here. The documentation is very hard to follow so I will summarize it.

1. The first step was to add the functions into its own file named WhateverMyOperationNameIs.js within the src/core/operations folder.

2. After this I needed to create another file also called WhateverMyOperationNameIs.jsin the src/core/config/modules/ folder. The code in here was simply importing the code from the file above and referencing these functions with different names.

3. .Then I needed to find a file called OpModules.js located in the src/core/config/modules/ folder. In here I had to import the module I wrote in the file above.

4. Then I needed to find a file called OperationsConfig.js in the src/core/config file to add a description of the parameters, return types, function purpose of the functions I added.

5. After that I needed to add my functions to the menu portion of the CyberChef interface. This was done via the src/core/config/Categories.js file.

6. After this I needed to add my tests to the repository so I created my own file in the src/test/tests/operations folder named the same as step 1 and 2.

All of these steps were new to me because I haven’t been asked to create modules, add modules to a module builder, add my operations, add it to an operation caller, and etc. These were definitely learning curves that I was able to overcome because of observing the code done by the previous contributors. I used a lot of their files to understand the syntax and the purpose of these files wherever I stumbled onto in the repository.

After doing all these steps then I finally ran the grunt dev and the grunt test commands. They were successful meaning I can submit my Pull Request. :) I then submitted my pull request and the next step from here was waiting for the admins to get back to me on my code.

Then I quickly realized my job wasn’t done and it was because of travis.ci.I quickly went from being someone that knew a little about travis.ci to being someone who avidly hates it in the matter of 30 minutes. Travis.ci is used to make small code reviews on github pull requests to identify errors that humans don’t need to. In this case travis was executing eslint and this identified several styling errors I had by using tab instead of spaces for code indentation.

This led to me making several commits to solve these errors, just to be greeted with another set of styling errors. After the 6th commit, travis.ci identified two major errors that I made and that was not including my changes to the package.json and package.lock.json file for installing the csv-string library, and my test results were not correct. After fixing these errors, on the 8th commit travis decided to give me the green checkmark and my Pull Request was finally ready. Although I do not like travis.ci because of how annoying it was, I can see that is definitely useful and is crucial in repositories to help screen pull requests before the admins invest any time into reviewing them.

Reflection

These bug fixes thought me a lot about how different open source repositories contributions work as well as different formats for submitting pull requests. Behind these two successful bugs were numerous other bugs that I could not fix simply because my environment could not be set up properly or the issue was very daunting after digging into it a little.

One thing I did learn from both bugs is that small bug fixes are just as impactful as big bug fixes since you are usually improving an existing functionality or fixing a broken one. Handling minor bug fixes requires digging into the code much deeper than just scanning the face of the project, which is what I have done before while casually scrolling through GitHub, and this has built up a lot of curiosity in me to do more of it in the future.

All in all these bug fixes thought me a lot about how to contribute to the open source community and now its time for me to do much much more. See you on my next blog post!